Overview — Measuring Test Quality Honestly

By now the book has argued, page after page, that you cannot prove software correct — you can only buy justified confidence and spend your effort where the risk is highest. That raises a question we have so far been able to dodge: once you have written tests, how do you know whether they are any good?

“We have tests” is not an answer. A suite of a thousand tests that never touch your payment path is worse than three tests that do, because the thousand look like safety while providing none. So the moment you take testing seriously, you need a way to look at a suite and judge it — to turn a vague feeling (“I think we’re covered”) into something you can point at, compare across releases, and argue about in a review. That is what this Part is: a tour of the ways people measure test quality, and — just as important — the ways each of those measures lies.

The question this Part answers

Every page here serves one question:

How do you measure whether your tests are any good — and why is “we have tests” not an answer?

The honest reason we measure at all is that confidence does not scale by intuition. On a ten-line function you can read the tests and know whether they exercise the risky paths. On a million-line system with fifty contributors, nobody holds the whole thing in their head. You need a proxy — a number that stands in for “are the dangerous paths exercised?” so that a human, or a CI gate, can make a decision without re-reading everything.

That is the entire promise of a test-quality metric: it compresses “is this suite good?” into something cheap to compute. And that promise is also the trap.

The core tension: a metric is a proxy, never the thing

Hold this sentence for the whole Part, because everything else hangs off it:

A metric is a proxy for confidence. It is never confidence itself.

Coverage is not correctness. A defect count is not quality. “95% line coverage” is a fact about which lines your tests executed — it says nothing about whether the tests checked the right thing when they executed them (recall the oracle problem). You can reach 100% coverage with tests that assert nothing at all.

   what you actually want          what the metric reports
   ──────────────────────          ───────────────────────
   "the risky behaviours are   ≠   "95% of lines ran during
    checked, and a bug in           the test suite"
    them would fail a test"
                                    (the gap between these two
                                     is where false confidence lives)

The gap between the two columns is the subject of this entire Part. A good tester reads every metric as a question, not an answer: “97% coverage — of what, and what is the 3% I’m not touching?” The number is where the investigation starts, not where it ends.

The three families of metrics we will study

The metrics people use to judge tests fall into three families, and this Part walks through them in order of how directly they probe your tests versus your product.

• Coverage metrics measure how much of the code your tests execute — in increasingly strict flavours: line, branch, condition, and path coverage. They are cheap, automatic, and the most widely trusted number in the industry. They are also the easiest to game, because executing a line is not the same as testing it.
• Mutation testing turns the microscope around and measures the quality of the tests themselves. It deliberately breaks your code (flips a < to a <=, deletes a line) and asks: did any test notice? A test that passes against broken code was never testing anything. Mutation testing is the closest thing we have to a direct measure of “would these tests catch a real bug?”
• Defect metrics measure the outcome in production — how many bugs escaped to users (escape rate), and how fast you recovered when they did (MTTR). These are the only metrics that measure reality rather than a proxy for it, but they are lagging: by the time they move, the bug has already reached a user.

Coverage looks at your code. Mutation testing looks at your tests. Defect metrics look at your users. No single one tells you the suite is good; read together, they triangulate.

The roadmap for this Part

Read these in order. Each page names both what its metric measures and — the part everyone forgets — what it cannot measure.

#	Page	What it measures	What it cannot measure
2	What Coverage Actually Measures	Which code your tests execute, and how a coverage tool instruments code to find out	Whether the executed code was actually checked by an assertion
3	Coverage Types — Line, Branch, Condition, Path	The strictness ladder: statements, then branches, then boolean sub-conditions, then whole paths	Diminishing returns — path coverage is usually infeasible, so you always stop short of “everything”
4	Why 100% Coverage Is Not Bug-Free	The ceiling of coverage: it proves lines ran, nothing more	The bugs living in inputs, states, and missing code that coverage structurally cannot see
5	Mutation Testing — Testing Your Tests	Whether your tests would catch a change — the strongest proxy for real fault-finding power	Semantic bugs no small mutation models, and the cost (it is slow and noisy)
6	Defect Metrics — Escape Rate and MTTR	Reality: bugs that reached users, and how fast you recovered	Anything before the bug ships — it is a lagging, after-the-fact signal
7	Goodhart’s Law — When a Metric Becomes a Target	The failure mode common to all the above: a metric optimised as a goal stops measuring anything	Itself — which is the whole point
900	Revision — Coverage & Quality Metrics	A prose recap tying every metric back to the proxy-not-confidence tension	—

The order is deliberate. Coverage first, because it is what almost everyone reaches for and where the false confidence starts. Then mutation testing, which exists precisely to expose coverage’s biggest lie. Then defect metrics, which check whether any of it worked in the real world. And finally Goodhart’s law, because by then you will have seen four metrics, and the one lesson that unites them is what happens when you turn any of them into a target.

The warning that will close the Part

Every metric in this Part shares a single failure mode, and we save its full treatment for the last page — but it is worth planting now, because it should colour how you read everything in between:

Goodhart’s law: when a measure becomes a target, it ceases to be a good measure.

The instant your team is told “get coverage to 90% or the build fails,” coverage stops measuring test quality and starts measuring compliance. People write assertion-free tests that execute lines without checking them; they delete hard-to-cover error handlers; they mark risky code as excluded. The number goes up. The confidence it was supposed to represent goes down. Every metric here can be gamed this way, and the more mechanical the reward, the faster it happens.

It actually happened — the metric-as-target pattern

Across the industry there is a well-documented, recurring pattern: an organisation mandates a coverage threshold as a hard release gate, and within a few quarters the coverage number is high and stable while production defects are unchanged or worse. Post-mortems repeatedly find the same mechanism — teams wrote tests engineered to touch lines rather than to check behaviour, sometimes with no assertions at all, purely to clear the gate. The dates and companies vary and the specifics are easy to get wrong, so treat any single retelling as illustrative; the pattern itself is robust, and it is exactly Goodhart’s law operating on a testing metric. The lesson: a threshold you cannot game is rare, so measure the thing you care about (did bugs escape?) alongside the proxy, and never let the proxy alone decide.

The thread

This Part pulls hard on the book’s throughline:

How does this technique buy me more justified confidence per unit of effort — and which edge cases does it flush out before my users hit them?

A metric is only worth computing if it changes a decision you would otherwise get wrong. Coverage that flags an entirely untested module buys confidence cheaply. Mutation testing that reveals your tests assert nothing buys a lot of confidence for a lot of compute. But a metric you chase for its own sake buys negative confidence — it manufactures the feeling of safety without the substance, which is worse than no metric at all, because it stops you looking.

   the metric's job                    the metric's trap
   ────────────────                    ─────────────────
   turn "is this suite good?"          become the goal, so people
   into a cheap, comparable      vs.   optimise the number instead
   number a human can act on           of the confidence it stood for
        │                                        │
        └──────────── same number ───────────────┘
              (which one you get depends entirely
               on whether you made it a target)

Read every metric in this Part as a witness to be cross-examined, never a verdict to be trusted. When we reach real code — for example asking whether the kvlite server tests in rust/kvlite/tests/server.rs merely run the SET/GET handlers or actually assert that GET name returns exactly what was stored — the whole Part comes down to the difference between a line that executed and a behaviour that was checked.

Check your understanding

1. The Part’s opening claim is that “we have tests” is not an answer to “are the tests any good?” Why not — what can a large suite have and still be worthless?
2. State the core tension of this Part in one sentence, using the word proxy.
3. Name the three families of metrics this Part covers and, for each, say whether it primarily inspects your code, your tests, or your users.
4. Coverage can reach 100% on a suite that finds no bugs. Explain, using the oracle problem, how that is possible.
5. State Goodhart’s law and give a concrete example of it acting on a coverage threshold.

Show answers

1. Because quantity is not quality: a suite can have thousands of tests that execute lots of code while never checking the behaviours that matter (e.g. never touching the payment path, or asserting nothing). Such a suite looks like safety and provides none, which is worse than a small suite that exercises the risky paths, because it hides the gap.
2. A metric is a proxy for confidence; it is never confidence itself — so the number can move while the thing it stands for moves the opposite way.
3. Coverage metrics (line/branch/condition/path) inspect your code — which lines ran. Mutation testing inspects your tests — whether they would notice a deliberate bug. Defect metrics (escape rate, MTTR) inspect your users / production reality — what actually escaped and how fast you recovered.
4. Coverage only records that a line executed during a test run. It says nothing about whether the test had an oracle — an assertion checking the result was correct. A test with no assertions (or a wrong one) can run every line and pass regardless of the output, so you can execute 100% of the code while verifying none of it. Coverage measures execution; correctness needs an oracle.
5. Goodhart’s law: when a measure becomes a target, it ceases to be a good measure. Example: mandate “90% line coverage or the build fails,” and teams write assertion-free tests that touch lines without checking them, or exclude hard-to-cover code — the coverage number rises to satisfy the gate while the confidence it was supposed to represent falls.