Revision — The Levels of Testing

This part took the book’s one question — how do you earn justified confidence that software works? — and gave it its first structural answer: you earn confidence at different levels of scope, and the levels are not interchangeable. We opened by noticing that a test always fixes two things before it runs — how much of the system it exercises, and what it lets stand in for the rest — and everything after followed from that. This page walks back through the whole arc so the ideas sit as one shape rather than seven pages.

The five levels, and the confidence each buys

The overview laid the spectrum from a single function in isolation out to a whole product a user sees, and named five useful points on it. Each level buys a different currency of confidence, and no amount of one currency converts into another.

• Unit tests pin one function or class with its collaborators faked out. A green unit test says this logic is right — the branch, the math, the off-by-one. It is fast (sub-millisecond), you keep thousands, and a failure points at one place. What it misses: whether the unit is ever called, whether it is wired to anything real, whether the fakes it trusts match reality.
• Integration tests wire a few real parts together and put the seam between them under test. A green integration test says these two real parts agree — the serializer and the parser share a format; the service and the database speak the same SQL. What it misses: everything above the parts it wired, and any seam it still faked.
• System tests boot one whole deployed service and drive it through its real edges. A green system test says the assembled thing works — real I/O, real config, real startup. What it misses: the parts that only exist when the product meets its real neighbours.
• End-to-end (E2E) tests drive the whole product plus its real dependencies like a user would. A green E2E test says a real user journey works start to finish. It is slow, costlier, and prone to flake, so you keep few — but a pass means a great deal, because most of the real system was involved. What it misses: whether that journey is the one the customer actually needed.
• Acceptance tests check the product against the requirement, not the code. A green acceptance test says this is the right thing — fitness for purpose, not just correctness. What it misses: nothing about the requirement, but everything about internal quality — a feature can be the right one and still be a mess inside.

The jump from the first four to acceptance is the one to hold onto. The first four all ask does it work? — correctness. Acceptance asks is it the thing we needed? — fitness for purpose. A system can pass every correctness level and still fail acceptance because it flawlessly implements the wrong thing. That “built the wrong thing, correctly” bug is the most expensive one there is, because everything downstream of it passed.

  UNIT ───► INTEGRATION ───► SYSTEM ───► E2E ───► ACCEPTANCE
   │             │             │          │            │
 "is this     "do these     "does the  "does a     "did we build
  piece        parts agree   assembled  real user    the RIGHT
  right?"      at the seam?"  thing      journey      thing?"
                              work?"     work?"
   └──────── correctness: does it work? ────────┘   └─ fitness ─┘

The trade that shapes every suite

The reason there are levels at all — and the reason you cannot just write the most realistic test every time — is a single trade that runs the length of the spectrum. As a test’s scope grows, it climbs in fidelity (how much real system it proves) but also in cost, runtime, and flakiness, so you can afford fewer of them.

  ◄── faster · cheaper · more of them · precise failure ──
  UNIT            INTEGRATION         SYSTEM/E2E        ACCEPTANCE
  -- slower · costlier · fewer of them · higher fidelity ──►

A unit test runs in milliseconds and fails at one line; an E2E test runs in seconds, touches config and timing and the network, and when it fails you may spend an afternoon finding out where. That is not a defect of the higher levels — it is the price of the realism that makes their pass mean more. The “By the numbers” note in the overview made this concrete: on the same wall-clock budget you can run roughly two hundred unit tests for every one E2E test. That ratio is not a slogan — it is the entire engine behind the debate over suite shape.

Pyramid vs trophy: the debate is about ratio

Given that trade, the open question is: how many tests do you keep at each level? The pyramid-vs-trophy page laid out the two dominant answers.

• The testing pyramid says keep a broad base of cheap unit tests, fewer integration tests above, and a thin cap of E2E tests. It optimizes for a fast, cheap, non-flaky suite with pinpoint failures. Its failure mode is a base that proves little — thousands of green unit tests over heavily-mocked seams, while the real bugs live in the wiring the mocks papered over.
• The testing trophy re-weights toward the integration middle, arguing that in modern service- and component-heavy code most real bugs live at the seams, so that is where the tests should concentrate. Its failure mode is a slow, expensive middle if you push integration where a unit test would have done.

The lesson was not that one shape wins. It is that the right shape follows where your bugs and your costs actually live. A pile of pure-logic code with few collaborators earns a pyramid honestly; a thin service that is mostly glue between a database, a queue, and an API earns a trophy honestly. The shapes are named hypotheses about where bugs hide — you pick the one that matches your codebase, and you revise it from your own failure data, not from a diagram on a slide. A shape you cannot defend with “this is where our bugs were” is cargo cult.

The composition rule: lowest honest level wins

Composing the levels into a strategy turned the ratio into a rule you can apply test by test. State it as one sentence and it does most of the work:

Test each behaviour at the lowest level that can honestly prove it, and let the higher levels cover only the gaps the lower ones structurally cannot reach.

“Lowest honest level” carries the weight. Push a check as far down as it will truthfully go — a unit test if the logic can be pinned without lying about its collaborators, an integration test the moment the behaviour lives in a seam a fake would have to invent. “Honestly” is the guard against the pyramid’s failure mode: a fast test that only passes because its mocks agree with themselves is not proof, it is theatre. When a behaviour cannot be proven honestly at a level, it earns promotion to the next level up — and only then.

The payoff is a suite with no accidental redundancy and no silent gaps: each class of bug is caught once, at the cheapest level that can catch it for real, and the expensive higher levels spend their small budget only on the confidence that genuinely lives up there — real wiring, real journeys, the right requirement. You are not stacking five tests on one bug while another class goes untested; you are spending a fixed confidence budget where it buys the most.

  behaviour ──► can a UNIT prove it honestly? ── yes ─► test here, done
                        │ no
                        ▼
                 can INTEGRATION? ────────────── yes ─► test here, done
                        │ no
                        ▼
                 SYSTEM / E2E / ACCEPTANCE ───────────► test here (few, precious)

You saw this rule as living code, not theory. The book’s two small Rust projects already embody it: kvlite’s pure logic is pinned by fast unit tests, while its tests/ directory holds exactly the integration-level checks that a unit test couldn’t honestly make — for example a concurrency test that hammers one shared store from many threads, and a server test that drives the real TCP socket end to end. Each of those lives one level up precisely because the behaviour it proves — real concurrency, a real socket — cannot be faked without lying.

The thread this part fed

Zoom back out to the book’s one question. This part’s answer is that confidence is layered, and the engineering skill is composing the layers, not crowning a favourite. A microscope is not better than a telescope; unit tests give you speed and precision, acceptance tests give you meaning, and the levels between stitch those extremes into one whole. Each level catches a class of bug the others structurally cannot, so the goal was never the “best” brick — it was the finished stack.

That is what “justified confidence that software works” cashes out to in practice: not a green suite of any single kind, but a deliberately composed set of levels where you can point at any class of failure and say which level would catch it, why it lives there, and what it would cost you to catch it anywhere else. Carry that composing instinct forward — the parts ahead push on the harder half of the book’s question, the edge cases that break software even when every level above is green.