Revision · Coverage & Quality Metrics

The overview opened this Part with one sentence you were asked to hold the whole way through: a metric is a proxy for confidence, never confidence itself. Every page since has been that sentence stress-tested against a different number — coverage, mutation score, escape rate, MTTR — and the pattern held every time. The number is where an investigation starts, not where it ends.

This page walks back over the Part with that discipline in hand. Nothing below is new. It is the six content pages compressed, kept at the level of certainty each deserves, so the numbers resolve back into the book’s single question: how do you earn justified confidence that software works — rather than manufacture the feeling of it?

Coverage measures execution, not correctness

Start where almost everyone starts, because it is where the false confidence starts too. What coverage actually measures is one thing and one thing only: which lines of your code ran while the test suite executed. A coverage tool instruments your code — it inserts invisible counters at each line or branch — runs the suite, and reports which counters fired. That is the entire mechanism, and it is worth saying plainly what it does not touch: the assertions.

Coverage cannot see whether a line that ran was ever checked. A test with no assertions at all executes code exactly as thoroughly as a test with a dozen — the counters fire either way. This is the oracle problem wearing a metric’s clothing: executing a behaviour and verifying it are two different acts, and coverage only witnesses the first.

   what coverage records          what coverage cannot record
   ─────────────────────          ───────────────────────────
   this line ran                  ...and the result it produced
   during a test                  was the correct one
                                   (that requires an assertion —
                                    an oracle — which coverage
                                    never inspects)

Which is why 100% coverage is not bug-free, and never can be. Even a suite that runs every single line can miss three whole categories of bug that coverage is structurally blind to:

• Input bugs. A line that handles discount(price) runs fine on price = 100, hits 100% coverage, and still divides by zero on price = 0. Coverage counts lines, not the values that flow through them.
• State and ordering bugs. The line ran, but only in one order, one session, one configuration. Coverage has no notion of how many ways you reached that line.
• Missing-code bugs. The most brutal category: you cannot cover a line you never wrote. The forgotten else, the unhandled error case, the validation that was never added — coverage reports 100% because the buggy code does not exist to be measured.

The takeaway of the first three pages is a single reframing: read “97% coverage” as a question — “of what, and what is the 3%?” — not an answer. The 3% is often exactly the error-handling and edge cases where the real bugs live.

The coverage-type ladder: each rung sees more, none sees enough

Coverage types form a strictness ladder. Each rung is a stricter definition of “covered,” catches a class of bug the rung below it misses, and costs more to satisfy. Knowing the ladder is knowing precisely what your coverage number cannot promise.

   RUNG        "covered" means...              catches                      still misses
   ─────────   ────────────────────────        ────────────────────         ──────────────────────
   line        every statement ran             totally-untested code        untaken branches
   branch      every if-true AND if-false      the skipped else,            which condition inside a
               edge was taken                  the empty error path         compound test mattered
   condition   each boolean sub-expression     a && b where b was           which combination of
               was both true and false         never independently false    conditions actually fires
   path        every route through the         interactions between         combinatorial explosion —
               branches was walked             decisions in sequence         usually infeasible

The load-bearing detail is at the two ends. Line coverage is the weakest and most gameable: it is satisfied by a straight-line test with no branch decisions checked, so a line inside if (rare_condition) counts as covered the moment the function runs at all. Branch coverage — take both the true and false edge of every decision — is the honest working default, because most real bugs hide in the path you didn’t take, especially the error branch. Condition coverage goes finer, checking each sub-clause of a compound boolean like a && b independently, and catches short-circuit bugs branch coverage waves through. Path coverage is the theoretical ceiling — every combination of every decision — and it is almost always infeasible: a function with n independent branches has up to 2ⁿ paths, so the count explodes past any hope of enumeration. That infeasibility is not a footnote; it is the reason you always stop short of “everything,” and why coverage always leaves a gap.

By the numbers — why path coverage explodes

   A function with just 10 independent binary decisions:

   line coverage      : 1 test can hit every line          →  achievable
   branch coverage    : ~2 well-chosen tests per branch     →  ~20 tests, achievable
   path coverage      : 2^10 distinct routes                →  1,024 paths

   Add one more decision and path count doubles to 2,048.
   This is why teams target branch coverage and treat full
   path coverage as a theoretical ceiling, not a goal.

The lesson: pick the rung that matches the risk of the code. Branch coverage for ordinary logic; reserve exhaustive path thinking for the small, dangerous core where a missed interaction is catastrophic.

Mutation testing: the way to test your tests

Coverage grades your code’s exposure to the suite. It never grades the suite. That blind spot is exactly what mutation testing exists to fill, and it is the strongest proxy in the Part for the question that actually matters: would these tests catch a real bug?

The mechanism is beautifully direct. A mutation tool makes a tiny, deliberate change to your source — flips a < to <=, swaps + for -, deletes a line, replaces a boolean with true — producing a mutant: a version of your code with a planted bug. Then it runs your suite against the mutant and asks one question:

   plant a small bug in the code (a "mutant")
                 │
        run the test suite
                 │
        ┌────────┴────────┐
   a test FAILS       all tests PASS
        │                 │
   mutant "killed"   mutant "survived"
   (good — a test    (bad — a real bug of
    would catch      this shape would ship
    this bug)         with a green suite)

Your mutation score is the percentage of mutants your tests killed. A surviving mutant is a concrete, damning artefact: proof that a specific bug could be introduced and your suite would stay green. It is the direct answer coverage can only gesture at — and it exposes coverage’s biggest lie, because an assertion-free test that gives you 100% coverage kills almost no mutants. The mutations happen; the tests, checking nothing, never notice.

Mutation testing is not free, and the Part was honest about its costs: it is slow (the suite reruns once per mutant, so hundreds or thousands of times) and noisy (some mutants are “equivalent” — they change the code without changing its behaviour, so no correct test could ever kill them, yet they count against your score). That is why you point it at the small, high-stakes core rather than the whole codebase. But conceptually it is the honest one: it grades the tests, not the code’s exposure to them.

Defect metrics: the feedback production sends back

Coverage looks at your code. Mutation testing looks at your tests. Neither has ever met a user. Defect metrics are the only numbers in this Part that measure reality — what actually happened after the software shipped — and that is both their power and their limit.

Two matter most. Escape rate measures the bugs that got past every gate — the fraction of defects found in production rather than by your tests, roughly bugs_found_in_prod / (bugs_found_in_test + bugs_found_in_prod). It is the most honest report card your suite ever gets: a suite with high coverage and a high escape rate is a suite that runs a lot of code without checking the things users actually hit. MTTR (mean time to recovery) measures not whether you failed but how fast you recovered when you did — the clock from “a bug is affecting users” to “service restored.” It reframes the goal from the impossible (“never ship a bug”) to the achievable (“shrink the blast radius when you do”).

   coverage / mutation  ──►  measured BEFORE ship  ──►  leading, cheap, a proxy
   escape rate / MTTR   ──►  measured AFTER ship    ──►  lagging, real, the truth

   the two halves close the loop:
   escape rate tells you whether the pre-ship
   proxies were actually predicting anything.

The catch is right there in the diagram: defect metrics are lagging. By the time escape rate moves, the bug has already reached a user; by the time MTTR is measured, the outage already happened. They cannot prevent the incident you are measuring — but they are the ground truth that tells you whether your leading proxies were worth trusting. A suite whose coverage is high and whose escape rate is also high has just been caught lying, and only the production-side number could catch it.

Goodhart’s law: the failure mode every metric shares

Set the four metrics side by side and one danger unites them. The moment any of them becomes a target rather than a signal, it stops measuring what you cared about. This is Goodhart’s law, the warning the overview planted and the Part saved for last:

When a measure becomes a target, it ceases to be a good measure.

The mechanism is human, not mathematical. Tell a team “coverage must hit 90% or the build fails,” and coverage stops measuring test quality and starts measuring compliance. People do the rational thing under the incentive you actually set: they write assertion-free tests that touch lines without checking them, delete or exclude the hard-to-cover error handlers, and mark risky code out of scope. The number climbs to clear the gate. The confidence it was supposed to represent falls. And it happens fastest wherever the reward is most mechanical — a hard CI threshold gamed within a quarter.

Every metric here is gameable this way. Mandate a mutation score and people delete the mutants they can’t kill. Reward low escape rate and people reclassify production bugs as “feature requests.” Reward low MTTR and people close incidents before they are truly fixed. The metric is never the enemy; the target is.

It actually happened — the metric-as-target pattern

Across the industry there is a well-documented, recurring pattern: an organisation makes a coverage threshold a hard release gate, and within a few quarters the coverage number is high and stable while production defects are unchanged or worse. Post-mortems keep finding the same mechanism — tests engineered to touch lines rather than check behaviour, sometimes with no assertions at all, written purely to clear the gate. The exact companies and dates vary and are easy to get wrong, so treat any single retelling as illustrative; the pattern itself is robust, and it is Goodhart’s law operating on a testing metric. The durable fix is not a better single number but a pair: watch a leading proxy (coverage, mutation score) alongside a lagging truth (escape rate), so that gaming one shows up as a divergence in the other.

The guide-not-goal principle

Which resolves the whole Part into one working rule. A metric is a guide, not a goal. Its job is to start an investigation — to point at the module nobody tested, the mutant nobody killed, the bug pattern that keeps escaping — and hand a human a question worth asking. Its job is never to end one by declaring the suite good because a threshold was cleared.

Held as guides, the four metrics triangulate, because each covers the others’ blind spots:

   COVERAGE     ─► what code did the suite touch?      (cheap, gameable, blind to assertions)
   MUTATION     ─► would the suite catch a real bug?   (honest, slow, blind to semantic bugs)
   ESCAPE RATE  ─► what got past everything to users?  (true, lagging, blind to what didn't ship)
   MTTR         ─► how fast did we recover?            (true, lagging, measures response not prevention)
                        │
                        └─ no single one is trustworthy alone;
                           read together they cross-examine each other

The instant you promote any one of them from guide to goal — from a witness you cross-examine to a verdict you trust — Goodhart’s law converts it from a source of confidence into a source of the illusion of confidence, which is strictly worse than no metric at all, because it stops you looking.

The thread

This Part pulled on the book’s throughline from a specific angle: honest metrics are how you earn justified confidence rather than manufacture it. Every earlier Part taught you how to build confidence — good tests, the right test at the right level, thinking hard about edge cases. This Part taught you how to check whether that confidence is real, and it closed with a warning about the one move that converts real measurement into theatre.

Set beside the book’s central sentence — you cannot prove software correct; you can only buy justified confidence and spend your effort where the risk is highest — the metrics are simply the instruments that keep you honest about how much confidence you have actually bought. Coverage tells you where you spent effort. Mutation testing tells you whether that effort could catch anything. Defect metrics tell you what it missed in the real world. And Goodhart’s law is the reminder that the instruments only work while you are reading them, not gaming them.

Read every metric as a witness to be cross-examined, never a verdict to be trusted. When you reach real code — asking whether the kvlite server tests in rust/kvlite/tests/server.rs merely run the SET/GET handlers or actually assert that GET name returns exactly what was stored — the whole Part comes down to the same difference it opened with: a line that executed versus a behaviour that was checked. Justified confidence lives entirely in that gap, and honest metrics are how you keep watch over it.

To revisit any thread, return to the pages that prove it: what coverage measures, the coverage types, why 100% is not bug-free, mutation testing, defect metrics, and Goodhart’s law — this time reading each number aloud as a question, never an answer.