Defect Metrics — Escape Rate and MTTR
Every metric so far in this Part has looked inward, at the code and the tests. Coverage counts what ran; mutation score counts what your tests would catch. Both are measured on your machine, before anything ships. They are leading indicators — cheap, fast, and entirely under your control.
This page turns the telescope around. The ultimate judge of a test suite is not a number in a CI log; it is what happens when real users touch the software. A defect that no test caught, that no coverage report flagged, that reached production and bit someone — that is the verdict that matters. So we measure two outcome metrics: how many bugs escaped to users, and how fast we recovered when they did. Together they read your entire testing process as a black box and grade it on results.
Defect escape rate — the bugs that got past you
Section titled “Defect escape rate — the bugs that got past you”A defect escape is a bug that was not caught before release and was instead discovered in production, by users, monitoring, or support. The defect escape rate is the fraction of all defects that escaped:
defect escape rate = defects found in production / total defects found
found before release: 84 found in production: 16 ---------------------------- total: 100 escape rate = 16 / 100 = 16%Read it plainly: of every 100 real defects in this release, 16 slipped past every test, review, and gate you have, and a user found them first. It is the most honest single number about your testing process, because it counts only the failures that actually reached the outcome you were trying to prevent. Coverage can lie to you (a covered line is not a checked line). Escape rate cannot — a bug in production is a bug in production.
The denominator matters. Escape rate is a ratio, not a raw count. If you ship more features, you will find more defects everywhere; the raw number of escapes rising might just mean you shipped more. The fraction controls for that: it asks what share of your defects your process caught, independent of how much you shipped.
pre-release net ┌─────────────────────────────┐ (tests, review, │ caught 84 │ missed 16 │──► escaped to users CI gates) └─────────────────────────────┘ ▲ ▲ your suite the escape rate did its job measures this leakWhy escape rate complements the code-side metrics
Section titled “Why escape rate complements the code-side metrics”Coverage and mutation score tell you about the tests you wrote. Escape rate tells you about the tests you should have written and didn’t — the entire category of bugs your imagination never reached. That is a different, larger space.
METRIC MEASURED ANSWERS ───────────────────────────────────────────────────────────── line/branch cov pre-release "did a test run this code?" mutation score pre-release "would a test catch a broken line?" defect escape post-release "did real bugs reach real users?" MTTR post-release "how fast did we recover when one did?"The first two are leading indicators: you can improve them today and hope outcomes follow. The last two are lagging indicators: they tell you what your process actually delivered, but only after the fact. You need both. Leading metrics without lagging ones is confidence with no reality check — a 95% mutation score means nothing if escapes are climbing. Lagging metrics without leading ones is scorekeeping with no steering wheel — you know you’re losing but not which test to write. Escape rate closes the loop that coverage opened: coverage says “this code ran under test,” escape rate says “and here is what that testing was worth in production.”
Mean-time-to-repair — recovery as a first-class metric
Section titled “Mean-time-to-repair — recovery as a first-class metric”Prevention is only half the story. The tester’s instinct is to stop every bug before release, but no suite catches everything, and a process that can only prevent is brittle: the first escape it fails to stop becomes a prolonged outage. So we measure the other half — how fast you recover — with mean-time-to-repair (MTTR): the average time from a defect being detected in production to it being resolved (fixed, rolled back, or otherwise made safe for users).
defect detected ──────────────────────────► defect resolved │ │ │◄────────────── time to repair ──────────►│ │ │ └─ alert fires / user reports fix deployed or rollback done
MTTR = mean( repair time over many incidents )Why does recovery speed matter as much as prevention? Because the cost of an escaped defect is not fixed — it is roughly proportional to how long it stays live. The same bug caught and rolled back in 4 minutes versus one that festers for 4 hours does wildly different amounts of damage, even though the code is identical. A low MTTR turns a potential catastrophe into a footnote. This is the deep reason mature teams invest in fast rollback, feature flags, and observability: they accept that some defects will escape and make the escape cheap. Prevention lowers the rate; fast repair lowers the cost per event. Multiply them and you get the real exposure:
expected damage ≈ escape rate × incidents × (cost per minute × MTTR) └─ prevention ─┘ └──── recovery ────┘You can attack that product from either side. A team obsessed only with prevention has driven the left factor down and ignored the right — until one escape it never imagined runs for hours. A resilient team drives both down.
Gathering these honestly
Section titled “Gathering these honestly”Both metrics are only as good as the bookkeeping behind them, and the bookkeeping is where teams quietly lie to themselves. Two disciplines make them trustworthy.
Tag every defect with where it was found. When a bug is filed, record its phase of discovery: unit test, integration test, code review, staging, or production. This single field is the entire basis of escape rate — without it you cannot separate “caught” from “escaped.” Make it a required field, and make the categories unambiguous.
FOUND-IN COUNTS AS ────────────────────────────────── unit test caught (pre-release) integration test caught (pre-release) code review caught (pre-release) staging / QA caught (pre-release) ────────────────────────────────── production ESCAPED ◄── the only bucket that hurtsTimestamp detection and resolution. MTTR needs two honest clocks: when the defect was detected in production (first alert, first user report) and when it was resolved (fix or rollback live for users). Resist two temptations: starting the clock late (at “engineer started working,” not at “alert fired”) flatters your MTTR by hiding the time nobody noticed; stopping it early (at “PR merged,” not “deployed and verified”) hides deploy latency. Measure detection-to-recovery as the user experienced it, or the number is theatre.
Under the hood — deriving both from one incident log
Section titled “Under the hood — deriving both from one incident log”You do not need special tooling; a single append-only record per defect yields both metrics. Each row carries where it was found and, for production defects, two timestamps:
defect_id found_in detected_at resolved_at severity ───────────────────────────────────────────────────────────────────── D-1041 unit — — low D-1042 production 2026-07-02 09:14 2026-07-02 09:41 high D-1043 code_review — — med D-1044 production 2026-07-05 22:03 2026-07-05 22:12 high ...
escape rate = count(found_in == production) / count(all) MTTR = mean(resolved_at - detected_at) over production rowsThe same log answers other questions cheaply: escape rate by severity (are the escapes trivial or dangerous?), escape rate by module (which subsystem leaks?), and MTTR by detection source (do alerts recover faster than user reports — proof your monitoring works). One honest table, many honest answers.
Interpreting the trend — a rising escape rate is a design signal
Section titled “Interpreting the trend — a rising escape rate is a design signal”The mistake to avoid is reading a rising escape rate as “we need more tests.” More tests of the same kind that already missed these bugs will not catch the next ones — you will add coverage to code that was never where the escapes came from. A rising escape rate is feedback that your test design has a blind spot: some category of behaviour, input, or failure mode is systematically going untested.
So treat each escape as a probe. When one lands, ask the diagnostic question from the tester’s mindset: what class of test would have caught this, and why didn’t we have it? Then classify the escapes and look for the pattern:
escapes clustering in... the design gap is... ────────────────────────────────────────────────────────────── boundary/edge inputs missing boundary-value tests one module or service thin suite there; add focus integration seams too many unit tests, too few integration/contract tests concurrency / timing no stress/race tests (see kvlite's concurrency.rs — tests that exist precisely to find these) config / environment tests all pass in one env onlyEach cluster names a specific kind of test to add, not “more tests.” That is the whole value of the metric as feedback: it does not just tell you the suite leaked; the shape of the leak tells you which hole to patch. A flat-but-nonzero escape rate is a stable, understood process. A rising one is a warning that the world (new features, new inputs, new integrations) has outrun your test design — and no amount of re-running the tests you already have will fix it.
The architect’s lens
Section titled “The architect’s lens”- Why do these metrics exist? Because code-side metrics like coverage and mutation score are measured before release and can look perfect while real bugs still reach users. Escape rate and MTTR judge the testing process by its actual outcome in production — the only verdict that ultimately counts.
- What problem do they solve? They close the feedback loop from production back to test design (escape rate) and make recovery speed a first-class goal alongside prevention (MTTR), so a team optimises for both stopping bugs and surviving the ones that get through.
- What are the trade-offs? They are lagging indicators — you only learn from a defect after it has already hurt someone, and small denominators make short-term rates noisy. They also depend entirely on honest bookkeeping (where-found tags and detection/resolution timestamps), which is easy to fudge.
- When should I avoid them? Never fully — but don’t over-read a single release’s numbers on a low-volume service, and don’t use them as the only metrics: without leading indicators like coverage and mutation score you know you’re losing but not which test to write.
- What breaks if I remove them? You lose all sight of whether your testing actually works in the real world. Coverage stays green, confidence stays high, and a rising escape rate — the early signal of a test-design blind spot — goes unnoticed until an escape becomes an outage nobody was measuring.
Check your understanding
Section titled “Check your understanding”- Define defect escape rate precisely, including its denominator. Why is it expressed as a ratio rather than a raw count of production bugs?
- Escape rate and MTTR are called lagging indicators, while coverage and mutation score are leading. Explain the difference and why a team needs both kinds.
- “Prevention and recovery both reduce the damage from defects.” Using the expected-damage relationship in the page, explain how MTTR reduces exposure even when the escape rate is unchanged.
- What two pieces of bookkeeping make escape rate and MTTR trustworthy, and give one specific way each metric gets quietly fudged if you’re careless.
- Your escape rate rose from 12% to 20% over a quarter, with the escapes clustering at the seams between services. Why is “write more tests” the wrong response, and what does the clustering tell you to do instead?
Show answers
- Defect escape rate = (defects found in production) / (total defects found, pre-release + production). It is a ratio because raw escape counts rise simply when you ship more; the fraction controls for volume and isolates what share of defects your process caught, independent of how much you released.
- Leading indicators (coverage, mutation score) are measured pre-release and are under your control now, but only predict outcomes; lagging indicators (escape rate, MTTR) are measured post-release and report what the process actually delivered, but only after a defect has already reached users. You need leading metrics to steer (which test to write) and lagging metrics to check reality (whether the steering worked); either alone is blind in the opposite direction.
- Expected damage ≈ escape rate × incidents × (cost-per-minute × MTTR). MTTR sits in the recovery factor, so halving MTTR roughly halves the cost per escaped defect even if the same number of defects escape — the bug is live for less time, so it does less damage. Prevention shrinks how often you’re hurt; recovery shrinks how much each event hurts.
- (a) Tag every defect with where it was found (unit/integration/review/staging vs. production) — the basis of escape rate; fudged by mislabelling a production-found bug as “caught in staging.” (b) Timestamp detection and resolution as the user experienced them — the basis of MTTR; fudged by starting the clock at “engineer began work” instead of “alert fired,” or stopping at “PR merged” instead of “deployed and verified.”
- “Write more tests” usually means more of the same kind that already missed these bugs, which won’t catch the next ones. The clustering at service seams is a test-design signal: the suite is probably heavy on unit tests and thin on integration/contract tests. The right response is to add the specific missing category — integration and contract tests at those seams — not to pile more coverage onto code that wasn’t leaking.