Overview — Testing Across Boundaries

Every earlier Part of this book has looked inside a component. Unit testing pinned down single functions in isolation; test design taught you how to sample an infinite input space to find the edge cases that break one piece of code. You can do all of that perfectly and still ship a broken system. This Part explains why — and what to do about it.

Here is the scene that sets up the whole Part. Your unit suite is a wall of green. Coverage is high. Every function does exactly what its tests say it does. You deploy, and the system falls over anyway: the payment service sent a field the ledger service never learned to read; the cache returned a stale value the database had already changed; the third-party API rate-limited you at 3 a.m. and your retry loop hammered it into a longer outage. None of those bugs live inside a function. They live in the spaces between functions, services, and machines — and those spaces are exactly what unit tests cannot see.

The confidence leak is at the seams

Recall the book’s throughline: how do you earn justified confidence that software works? A unit test buys you confidence about one component behaving correctly given assumptions about everything it talks to. Integration and end-to-end tests exist to check those assumptions — to verify that when you bolt the correct pieces together, the joins hold.

Why do the joins fail so often? Because a boundary is where two mental models meet, and the two sides were usually built by different people, at different times, under different assumptions. One side thinks the timestamp is UTC; the other reads it as local. One side sends money in dollars; the other stores cents. One side retries on failure; the other is not idempotent, so a retry double-charges. Each component is individually correct against its own idea of the contract. The bug is that the two ideas disagree, and no test that stays inside one component can ever notice.

   Unit tests see this:          The bugs live here:

   ┌───────────┐                 ┌───────────┐   ┌───────────┐
   │  function │                 │ service A │──►│ service B │
   │  (green)  │                 │  (green)  │   │  (green)  │
   └───────────┘                 └───────────┘   └───────────┘
        ▲                                    ▲
   correct in                          the SEAM: wire format,
   isolation                           order, timing, failure —
                                       tested by nobody yet

The lesson to carry through this Part: green unit tests plus a red system is not a contradiction — it is the normal state of a codebase that has never tested its boundaries.

Where this Part sits in the pyramid

The levels of testing Part introduced the test pyramid. This Part lives in its upper two tiers. From the bottom up:

        /\        End-to-end   — few, slow, highest fidelity
       /  \                       (whole system, real deployment)
      /----\
     /      \     Integration  — some, medium speed
    /--------\                    (a few real components together)
   /          \
  /   Unit     \  Unit         — many, fast, lowest fidelity to prod
 /--------------\               (one component, everything else faked)

The shape is a deliberate economic argument, not an aesthetic one. As you climb, each test covers more of the real system (higher fidelity) but costs more to write, runs slower, and fails for more reasons unrelated to the bug you care about. So you want a broad base of cheap unit tests, a narrower band of integration tests over the risky joins, and a thin cap of end-to-end tests that prove the whole thing lights up. Invert the pyramid — mostly slow end-to-end tests — and your suite becomes so slow and flaky that people stop trusting it, which means it stops buying confidence at all.

The vocabulary, made precise

These words get used loosely in the wild. This book uses them precisely.

• System under test (SUT) — the thing whose behaviour a given test is actually making a claim about. Everything else in the test is scaffolding.
• Collaborators — the other components the SUT talks to: a database, another service, a queue, a clock. In a unit test you usually fake them; in an integration test you use some real ones.
• Integration test — exercises two or more real components together across a boundary, to check that they agree. “My service talks to a real Postgres and the SQL actually round-trips” is an integration test.
• End-to-end (E2E) test — exercises the whole system the way a user or client does, through its real entry point, with real (or realistic) dependencies behind it, ideally in a production-like deployment. “A browser clicks Buy and a row appears in the orders table” is E2E.

The dividing line is scope, and it is a spectrum, not a switch: how much of the real system is in the picture, and how much is faked.

What “a boundary” actually is

The word “boundary” is doing heavy lifting, so make it concrete. A boundary is any place where control or data crosses from one context into another that can fail or disagree independently:

Boundary	What crosses it	How it bites you
Process	function call → IPC / FFI	serialization, ownership, panics that don’t propagate
Network	request → response	latency, partial failure, retries, ordering, timeouts
Database	in-memory object → rows	schema drift, transactions, nulls, migrations, races
Third-party API	your call → their service	rate limits, version changes, downtime you don’t control

Every entry in that table is a place a unit test cannot reach and a real user can. The real Rust integration test in rust/kvlite/tests/server.rs crosses the network boundary on purpose: it binds a real TCP listener on port 0, spawns the actual server on a thread, connects a real client socket, and asserts on the bytes that come back — PING → PONG, GET name → NIL. Nothing is mocked, because the whole point is to test the wire, not the function.

// The SUT is the real server + the real socket. Nothing faked.
let listener = TcpListener::bind("127.0.0.1:0").unwrap(); // OS picks a free port
let addr = listener.local_addr().unwrap();
thread::spawn(move || { let _ = server::serve_on(listener, db); });

let stream = TcpStream::connect(addr).unwrap();
// ...drive it exactly like `nc` would, and assert on the reply line:
assert_eq!(round_trip(&mut reader, &mut writer, "PING"), "PONG");

The central tension: fidelity vs speed vs stability

Everything in this Part is a negotiation between three properties, and you cannot max all three at once:

• Fidelity — how closely the test resembles production. A real database and a real network are high fidelity; an in-memory fake is low.
• Speed — how fast the test runs and thus how often it can run. Milliseconds invite you to run on every save; minutes push you to run nightly.
• Stability — how reliably a passing system produces a passing test. A test that fails 1% of the time for no code reason is flaky, and flakiness silently destroys the confidence the test was meant to create.

              FIDELITY
                 /\
                /  \      pick a point in this triangle;
               /    \     you cannot sit in all three corners.
              /      \
        SPEED --------  STABILITY

Raise fidelity by using real dependencies and you usually pay in speed and stability. Buy back speed and stability with fakes and you lower fidelity — now the test can pass while production would fail. Every technique in this Part is a specific, defensible answer to “where in this triangle should this particular test sit?” There is no globally correct point; there is only the point that is right for the risk you are trying to retire.

Roadmap for this Part

Read these in order. Each page delivers one concrete technique for earning confidence at the seams, and each is a different answer to the fidelity/speed/stability question above.

#	Page	Technique it delivers
2	Integration Strategies: Big-Bang vs Incremental	How to wire components together for testing — and why “assemble everything, then test” fails
3	Contract Testing: Consumer-Driven Confidence	Catch boundary disagreements between two services without deploying both together
4	Real vs Fake Dependencies	Mocks, stubs, fakes, and real dependencies — choosing the fidelity dial per test
5	End-to-End Tests and Their Cost	The highest-fidelity test, what it uniquely proves, and why you keep it thin
6	Flaky Tests: Why They Break	Diagnose the specific causes — timing, order, shared state, the network
7	Curing Flaky Tests	Concrete cures: determinism, isolation, waiting on conditions not clocks
900	Revision — Testing Across Boundaries	A prose recap of the whole Part

The thread

Hold every page in this Part against one question, and it is the same question from page one of the book: does this test buy me justified confidence about a real failure mode, at a price I can afford to pay on every change? Integration strategy and contract testing decide what boundary a test covers. Real-vs-fake decides how faithfully it covers it. End-to-end decides how much of the system is in scope. And the flakiness pages defend the whole edifice — because a boundary test you no longer trust is worse than no test at all: it costs you time to run, time to re-run, and eventually the habit of ignoring red. Confidence you cannot trust is not confidence.

Check your understanding

1. Your unit suite is entirely green but production is broken. Explain, in the vocabulary of this Part, why that is not a contradiction and where the bug most likely lives.
2. Define “integration test” and “end-to-end test” and name the single dimension that distinguishes them.
3. Give two concrete examples of a “boundary” from the table, and for each name a failure mode a unit test cannot detect.
4. State the three-way tension at the heart of this Part and explain why you cannot maximize all three in one test.
5. Why does the test pyramid put few end-to-end tests on top rather than many, given that they have the highest fidelity to production?

Show answers

1. Not a contradiction: a unit test only checks one component given assumptions about its collaborators; it never checks the assumptions themselves. The bug lives at a boundary — a seam between components where the two sides disagree about wire format, ordering, timing, units, or failure handling. Each side is individually correct against its own idea of the contract; the contracts disagree.
2. An integration test exercises two or more real components together across a boundary to check they agree. An end-to-end test exercises the whole system the way a real user or client does, through its real entry point. The distinguishing dimension is scope — how much of the real system is in the picture (and how much is faked).
3. Any two, e.g. Network: retries against a non-idempotent endpoint cause double-processing — invisible to a unit test that never sends a second request. Database: a schema migration drops a column the code still writes — invisible to a unit test that fakes the store. (Process: a panic that doesn’t propagate across an FFI boundary. Third-party API: a version change or rate limit.)
4. Fidelity (resemblance to production) vs speed (how fast/often it runs) vs stability (how reliably a healthy system passes it). You cannot maximize all three because raising fidelity with real dependencies costs speed and stability, while buying back speed and stability with fakes lowers fidelity — so every test picks a point inside the triangle.
5. Because as fidelity rises, cost, runtime, and the number of unrelated reasons a test can fail (flakiness) all rise with it. A suite dominated by slow, flaky end-to-end tests runs rarely and gets ignored, so it stops buying confidence. A broad, fast unit base with a thin E2E cap gets you most of the coverage for a fraction of the cost and instability.