Revision — Testing Across Boundaries

The overview opened this Part with an uncomfortable scene: a wall of green unit tests, high coverage, every function proven correct in isolation — and a system that still falls over in production. The payment service sends a field the ledger never learned to read; the retry loop double-charges a non-idempotent endpoint; the cache serves a value the database already changed. None of those bugs live inside a function. They live in the spaces between functions, services, and machines. This Part was the tour of those spaces, and this page walks the tour back in one line.

It re-derives nothing — the seven pages before it did the proving. It re-connects them, so that by the end you can trace one unbroken thread from “here is a boundary two components share” all the way to “and here is a test I trust that proves they agree, at a price I can afford.” If any claim here feels thin, follow the link back to the page that earns it.

The boundary thesis, restated

The whole book aims at one target: how do you earn justified confidence that software works — and how do you think about the edge cases that break it? Unit testing answered that question inside a component. This Part answered it between components, because that is where the surviving bugs are.

Here is why they survive. A unit test buys confidence about one component behaving correctly given assumptions about everything it talks to. It never checks the assumptions. A boundary is exactly where two mental models meet, and the two sides were usually built by different people, at different times, under different assumptions. One side thinks the timestamp is UTC; the other reads it as local. One side sends dollars; the other stores cents. Each component is individually correct against its own idea of the contract. The bug is that the two ideas disagree — and no test that stays inside one component can ever notice.

   Unit tests see this:          The surviving bugs live here:

   ┌───────────┐                 ┌───────────┐   ┌───────────┐
   │  function │                 │ service A │──►│ service B │
   │  (green)  │                 │  (green)  │   │  (green)  │
   └───────────┘                 └───────────┘   └───────────┘
        ▲                                    ▲
   correct given                       the SEAM: wire format,
   assumptions                         order, timing, units,
                                       failure — tested by nobody

The durable lesson from page one: green unit tests plus a red system is not a contradiction — it is the normal state of a codebase that has never tested its boundaries. Everything in this Part is a specific, defensible way to test one kind of seam.

The integration toolkit, as one coherent thing

The middle five pages look like five separate topics. They are really one toolkit, and it helps to see them as answers to three sequenced questions about a single boundary: how do I wire it, how faithfully do I fill it, and how much of the system do I put around it?

How to wire it: big-bang vs incremental

Integration Strategies answered the first question: once you have components that pass in isolation, how do you assemble them for testing? The tempting answer — big-bang, assemble everything and run the whole thing — is the wrong one, because when it fails (and it will) every one of the joins is suspect at once. A failure that could be anywhere is a failure you cannot localize, and localization is the entire value of a test.

The disciplined answer is incremental: add one component at a time, so each new failure implicates exactly the join you just made. That splits into two directions. Top-down starts at the entry point and works inward, filling not-yet-built lower layers with stubs — you exercise the high-level flow early but simulate the depths. Bottom-up starts at the leaf dependencies and works outward, using drivers to call up into the layers that do not have real callers yet. Neither is universally right; the choice is about which end carries more risk and which end is ready first.

   BIG-BANG                 TOP-DOWN                 BOTTOM-UP
   ────────                 ────────                 ─────────
   wire everything          entry point first        leaves first
   run it, hope             stub the depths          drive from above
   failure = anywhere       flow visible early        foundations solid
   cannot localize          needs stubs               needs drivers

The connective idea: stubs and drivers are the scaffolding that lets you test a join before both sides of it exist. That is the same move as a unit-test double, applied one level up — you fake what is not ready so the real thing under test is never blocked on it.

It is worth restating why big-bang is so seductive and so wrong, because the temptation never fully goes away. Big-bang feels efficient: you were going to assemble the whole system eventually, so why not test it all at once and skip the scaffolding? The answer is that a test’s value is not just “did it fail” but “did it tell me where.” Big-bang integration maximizes the first and destroys the second. When ten new joins light up red simultaneously, the failure carries almost no information — you are back to debugging by bisection, pulling components out one at a time until the culprit surfaces, which is exactly the incremental process you skipped, now run in reverse and under pressure. Incremental integration pays the scaffolding cost up front to buy localization, and localization is what turns a red bar into a fix instead of an investigation.

How faithfully to fill it: the real-vs-fake spectrum

Real vs Fake Dependencies answered the second question, and it is the hinge of the whole Part. For every collaborator across a boundary you set a fidelity dial, from a bare placeholder to the genuine article:

   LOWER FIDELITY  ◄─────────────────────────────►  HIGHER FIDELITY
   dummy    stub    fake    mock    test container    real dependency
   ─────    ────    ────    ────    ──────────────    ───────────────
   fills a  canned  working real     the real thing    the actual
   slot     answer  in-mem  spy w/   in a throwaway     prod service
            (input) impl    expects  Docker container

The vocabulary is the same one unit testing built — dummy, stub, fake, mock — now with two integration-scale additions. A fake is a real, working, lightweight implementation (an in-memory store) and is the workhorse of fast integration tests: high enough fidelity to catch real logic bugs, cheap enough to run constantly. A test container is the fidelity upgrade that made this decade’s integration testing practical — spin up the actual Postgres or Redis in a throwaway Docker container, test against it, tear it down. You get production’s real query planner, real transactions, real null handling, without a shared, stateful, hand-maintained test database.

The rule that ties the spectrum together: turn the dial no higher than the risk requires. A fake is enough to prove your logic; a real container is what you need to prove your SQL and your schema; a genuinely live third-party API is almost never worth its instability in an automated suite. Higher fidelity is not a virtue you maximize — it is a cost you pay only where a lower-fidelity double would let a real failure through.

There is a specific failure mode the dial exists to guard against, and it is worth restating because it is the whole reason “just fake everything” is wrong. A fake is a second implementation of the dependency’s behavior, written by you, from your understanding of it. Every place your fake and the real thing disagree is a bug your test cannot see — the fake says the query returns rows in insertion order, the real database makes no such promise; the fake accepts a null the real schema rejects. This is why the dial climbs to a test container at exactly the boundaries where your model of the dependency is the thing most likely to be wrong. You keep the fake for speed where the dependency’s behavior is simple and well-understood, and you pay for a real container where the dependency’s real, surprising behavior is precisely the risk you are trying to retire.

How much system to put around it: end-to-end

End-to-End Tests and Their Cost answered the third question by going all the way up. An E2E test exercises the whole system the way a user does — through the real entry point, with real dependencies behind it, ideally in a production-like deployment. A browser clicks Buy and a row appears in the orders table. It is the only test that proves the assembled, deployed system lights up: that config is wired, that the pieces you tested separately actually find each other, that the seams you could not see individually all hold at once.

That unique proof is expensive. E2E tests are slow (they wait on a real system), broad (they fail for dozens of reasons unrelated to your change), and the most flakiness-prone of all, because every real boundary they cross is a chance to time out, race, or drift. So the pyramid’s shape is not aesthetic — it is an economic argument. E2E tests are few but precious: you keep a thin cap of them over the handful of critical user journeys that must work, and you push everything else down to cheaper integration and unit tests. Invert the pyramid and the suite becomes so slow and flaky that people stop trusting it — at which point it has stopped buying confidence at all.

The repo’s own tests show where the line falls in practice. rust/kvlite/tests/server.rs is an integration test: it binds a real TCP listener, spawns the real server, and drives it over a real socket — one real boundary, the network, tested on purpose. rust/logwise/tests/cli.rs climbs one rung higher toward E2E: it invokes the built binary the way a user’s shell would, feeding real arguments and asserting on real exit codes and output. And rust/kvlite/tests/concurrency.rs proves a different seam entirely — it hammers one shared store from eight threads at once and asserts the final count is exact, which no single-threaded unit test could ever check. Three files, three boundaries — wire, process entry point, concurrency — each one a place a green unit test is blind. “Precious” means you write few of the slowest kind and make each one earn its runtime by proving something nothing cheaper can.

The trade that runs through all of it

Step back and every technique in this Part is one negotiation, the same triangle the overview drew: fidelity vs speed vs stability, and you cannot sit in all three corners.

              FIDELITY
                 /\
                /  \      every boundary test picks a point in here.
               /    \     no globally correct point exists —
              /      \    only the point right for the risk.
        SPEED --------  STABILITY

Fidelity — how closely the test resembles production. Real DB, real network: high. In-memory fake: low.
Speed — how fast, and thus how often, it can run. Milliseconds invite running on every save; minutes push it to nightly.
Stability — how reliably a healthy system produces a passing test. A test that fails 1% of the time for no code reason is flaky, and flakiness silently destroys the confidence the test existed to create.

Watch the trade move through the toolkit. A fake buys speed and stability by lowering fidelity — fast, deterministic, but it can pass while production fails. A test container buys back fidelity (real Postgres behavior) at a real cost in speed and a little in stability (a container has to start). An E2E test maxes fidelity and pays for it in both other corners at once.

That is why E2E is few-but-precious and the unit base is broad: the pyramid is just the fidelity/speed/stability triangle projected onto a whole suite, spending most of your test budget where speed and stability are cheap and reserving the costly high-fidelity corner for the few journeys that justify it.

Laid out side by side, the whole toolkit is one column of choices along that triangle — each technique a different, deliberate point, not a better or worse one:

Technique	Fidelity	Speed	Stability	Reach for it when
Fake / in-memory double	low–medium	fast	high	the dependency’s behavior is simple and the risk is your logic
Test container	high	medium	medium	your SQL, schema, or the dependency’s real quirks are the risk
Contract test	medium (per side)	fast	high	two services must agree but deploying both together is too costly
End-to-end	highest	slow	lowest	a critical journey must be proven on the assembled, deployed system

Read down the “reach for it when” column and the discipline is visible: you are always naming the specific risk first and then choosing the cheapest point on the triangle that retires it. There is no row that is universally best; there is only the row that matches the failure mode in front of you.

There is a fourth boundary technique in that table that deserves its own recap here, because it refuses the trade instead of accepting it.

Contract testing: catching disagreement without deploying both sides

Contract Testing attacks the seam between two services from a cleverer angle. The naive way to check that service A and service B agree is to deploy both and run an E2E test — slow, flaky, and it needs both teams’ whole worlds standing up at once. Contract testing splits that check in two so neither side ever has to meet the other in a test run.

The idea is consumer-driven: the consumer (service A) writes down exactly what it needs from the provider — these fields, these types, this shape — as a contract, and tests itself against a fake provider that honors that contract. The provider (service B) then takes the same contract and runs a test proving its real responses satisfy it. Two fast, independent tests, no shared deployment, and yet together they prove the boundary holds. The subtlety worth re-noting: it is consumer-driven on purpose. The contract records what the consumer actually uses, not everything the provider happens to return — so the provider stays free to add fields, change internals, and evolve, as long as it never breaks the specific promises a real consumer depends on. The contract is a precise, executable statement of the minimum the boundary must keep, which is exactly the disagreement that E2E tests catch late and expensively.

   CONSUMER SIDE                        PROVIDER SIDE
   ─────────────                        ─────────────
   "I need fields X, Y as ints"  ──►  the recorded contract
   test against a fake that              │
   honors the contract                   ▼
                                 "do my real responses
                                  satisfy that contract?"
   both pass  ⇒  the two agree, and neither deployed the other

The connective idea: contract testing is what you reach for when an E2E test would catch the disagreement but its price — the slowness, the flakiness, the dual deployment — is too high to pay on every change. It converts one expensive cross-boundary test into two cheap single-side tests that, taken together, make the same guarantee. It is the fidelity/speed/stability triangle outwitted by narrowing scope rather than lowering fidelity.

The pyramid is not a preference; it falls out of arithmetic. Put rough, illustrative per-test costs on each tier and the shape justifies itself. Figures vary wildly by stack — treat them as orders of magnitude, not measurements.

   tier          runtime/test   fidelity   flake-prone
   ────          ────────────   ────────   ───────────
   unit          ~1 ms          low        ~never
   integration   ~50–500 ms     medium     sometimes
   end-to-end    ~5–30 s        high       often

   a suite of 2,000 unit tests   ≈ 2 s      (run on every save)
   a suite of   200 integration  ≈ 40 s     (run on every push)
   a suite of    20 end-to-end   ≈ 400 s    (run before deploy)

   invert it — 2,000 E2E tests  ≈ 20,000 s  ≈ 5.5 hours, and flaky

An inverted pyramid does not just cost more wall-clock time; it multiplies the flake surface until the suite is red for reasons no one caused. A five-hour suite that is red 30% of the time for no code reason gets ignored, and an ignored suite buys zero confidence at any price. The shape is the cheapest arrangement that still covers the risk — most tests where speed and stability are nearly free, a precious few where fidelity is worth the cost.

Flakiness: the discipline that keeps the suite trustworthy

A boundary suite is only worth building if you trust its verdict, and every technique above adds real boundaries — real clocks, real networks, real shared state — which is exactly where trust leaks out.

The last two pages were the defense of the whole edifice, and they split cleanly into diagnosis and cure. That order is deliberate: a flaky test looks the same from the outside no matter what causes it — green sometimes, red sometimes, same code — so the cure you reach for is only as good as your diagnosis of the source. Guessing wrong (adding a longer sleep to fix what was really a shared-state leak) does not remove the flakiness; it just moves it. So the two pages are one loop: name the cause precisely, then apply the cure that removes that cause.

Flaky Tests: Why They Break named the causes. A flaky test is one that passes or fails on the same code — its verdict is noise, not signal — and the noise almost always comes from a small, nameable set of sources:

   CAUSE           WHAT LEAKS IN
   ─────           ─────────────
   Timing          sleep(500ms) that is 'usually' enough — until it isn't
   Order           test B passes only if test A ran first (hidden coupling)
   Shared state    a row, a file, a global left behind by the last run
   Network         real latency, rate limits, DNS, a third party at 3 a.m.
   Non-determinism the wall clock, a random seed, unordered map iteration

Curing Flaky Tests matched each cause with a cure, and the cures are one discipline wearing several hats: remove the source of nondeterminism, do not paper over it.

Isolation cures order-dependence and shared state: give every test its own fresh world — its own container, its own schema, its own temp dir — so nothing leaks between runs and no test can depend on another’s residue. The concurrency test above is a small worked example of the same instinct: it hands each thread its own key range so the final count is provable, rather than letting eight threads fight over shared keys and asserting on whatever survives. Isolation is not only between tests; it is the habit of never letting two things share state unless the sharing is itself the thing under test.
Wait, don’t sleep cures timing flakiness: never sleep a guessed interval and hope; poll for the condition you actually need (“the row exists”, “the server answers”) with a timeout. You wait exactly as long as reality takes, and no longer.
Control time and randomness cures nondeterminism: inject the clock and seed the RNG so “now” and “random” are values the test decides, not the machine’s mood. This is the same seam discipline from unit testing, applied to the two impurities that most often leak.
Quarantine, don’t blind-retry, is the meta-cure and the sharpest lesson of the Part. When a test flakes, the tempting fix is to auto-retry until it passes. That does not remove the flakiness; it hides it, and a hidden flaky test can now mask a real, intermittent production bug. The disciplined move is to quarantine the test — pull it out of the gating suite so it stops eroding trust in green — and then fix its root cause before it returns. Retry buys a green light by throwing away the signal the test existed to give.

Put the two pages together and the whole flakiness discipline is one lookup table — every cause has a matching cure, and the cure is always removal of a source, never suppression of a symptom:

Cause (why it flakes)	Cure (how to make it deterministic)
Timing — a guessed `sleep`	Wait on the condition, with a timeout — poll until true, not until the clock says so
Order — hidden test coupling	Isolate — each test gets a fresh world so none can lean on another’s residue
Shared state — leftovers	Isolate — own container, own schema, own temp dir; clean up or never share
Network — real latency / rate limits	Fake it where the network isn’t the SUT; where it is, wait and bound, don’t sleep
Nondeterminism — wall clock / RNG	Inject the clock, seed the RNG — make “now” and “random” values the test decides
A flake you can’t fix right now	Quarantine — pull it out of the gate; never blind-retry it green

The reason this matters is the reason it closes the Part. A flaky test is not a weak asset; it is a liability. Every false alarm spends the team’s attention and erodes the reflex to believe red — and the first time people learn to shrug at a red boundary test is the day the suite stops protecting the boundaries. Confidence you cannot trust is not confidence. Flakiness discipline is what keeps a boundary suite on the asset side of that line.

Under the hood — why boundaries leak nondeterminism in the first place

It is worth seeing why the causes above are exactly the ones that appear, because the pattern is not a coincidence — it is the price of admission for testing a real seam. A pure unit test is deterministic because it touches nothing outside itself: same inputs, same output, forever. Every step up in fidelity that this Part introduced deliberately reintroduces a piece of the outside world, and each piece brings its own clock or its own ordering:

   the moment you add...        you inherit its nondeterminism...
   ─────────────────────        ────────────────────────────────
   a real network call          latency, retries, partial failure, DNS
   a real database              transaction interleaving, row leftovers
   a real container             startup time you cannot predict exactly
   concurrency / threads        scheduling order the OS decides, not you
   the wall clock / RNG         a value that changes every run by design

Read that table next to the flakiness causes and they are the same list. Flakiness is not a separate hazard bolted onto integration testing; it is the shadow that every fidelity gain casts. That reframes the cures: you are not fighting bugs in your tests, you are paying back the determinism you spent to buy fidelity. Isolation buys back the determinism you lost to shared state; wait-don’t-sleep buys back what you lost to real latency; controlling the clock and seed buys back what you spent on the wall clock and RNG. Every cure is a targeted refund. Understanding the seam that leaked the nondeterminism is what tells you which refund to claim — which is why diagnosis (the causes page) has to come before the cure.

A recurring class of high-profile outages traces to a boundary that every side had tested in isolation and no test had exercised together under real conditions. A widely-reported financial-trading failure in the early 2010s, for example, involved deploying new code alongside an old code path that a since-repurposed flag re-activated; each piece behaved as its own tests expected, but the combination, live and at scale, sent a flood of erroneous orders and lost the firm hundreds of millions of dollars in under an hour. The specifics are firm-specific and best treated as directional, but the testing lesson is exactly this Part’s thesis: individually-green components are not a tested system. The seam — here, an old path meeting new config in a real deployment — was the one thing no unit or single-side test could see, and it is precisely what an incremental integration test, a contract check on the flag’s meaning, or a thin end-to-end smoke of the deploy path exists to catch. The gap between “each part passed” and “the assembled thing works” is not a rare edge; it is the default, and closing it is the whole job of this Part.

The whole thread, in one breath

Put end to end, this Part traces a single arc. The surviving bugs live at boundaries, because that is where two independently-correct components disagree. You test those boundaries by wiring components incrementally — top-down or bottom-up, with stubs and drivers as scaffolding — so each failure implicates exactly one join. You fill each collaborator to a chosen point on the real-vs-fake spectrum — dummy, stub, fake, test container, real — turning the fidelity dial no higher than the risk demands. You reserve the highest-fidelity, whole-system end-to-end test for the few precious journeys that must work, because it maxes fidelity and pays in speed and stability. Where an E2E check is too costly, contract testing splits it into two cheap single-side tests that agree without meeting. And over all of it you keep the verdict honest by hunting flakiness to its root — isolation, wait-don’t-sleep, controlled time and randomness, quarantine-not-retry — because a boundary test you no longer believe is worse than none.

That is what it means to earn justified confidence at the seams: not that the boundary tests exist, but that each one covers a real failure mode, at a fidelity the risk requires, at a price you can pay on every change, with a verdict you can trust. Unit tests proved the pieces. This Part proved they hold together.

One habit carries out of this Part and into everything after it. When you look at any system now, look first at its boundaries — the network hops, the database, the third-party call, the queue, the deploy step — and ask of each one: which side owns the contract, how faithfully am I testing that they agree, and how would I know, on every change, if they stopped agreeing? A codebase that can answer that for every seam is a codebase whose green suite means what it says. The next Part turns from the seams between components to the jagged inputs that break a single one — the empty, the enormous, the malformed, the adversarial — but the reflex is the same: confidence is earned one specific failure mode at a time, at the lowest price the craft allows.