Goodhart's Law — When a Metric Becomes a Target
The whole Part has circled one warning, and now it is time to say it plainly. The overview planted the seed — a metric is a proxy for confidence, never confidence itself. Coverage, mutation score, and escape rate each measured something real. And each, we noted in passing, could be fooled.
This page is about why they can be fooled, and it is not a quirk of any one metric. It is a law. The moment you take any of these numbers and make it a target — a bar to clear, a gate that blocks a merge, a line on someone’s performance review — you change what people optimize for. They stop chasing the thing the number was standing in for, and start chasing the number. And a number that people are chasing directly stops telling you the truth.
Goodhart’s law
Section titled “Goodhart’s law”The economist Charles Goodhart made the observation in the context of monetary policy in 1975; the crisp form we use today is usually credited to the anthropologist Marilyn Strathern:
When a measure becomes a target, it ceases to be a good measure.
Read it slowly, because the sting is in the causality. The metric does not merely become useless when you target it — it becomes actively misleading, and it does so because you targeted it. Before it was a target, coverage correlated with “the risky paths are exercised.” After it is a target, that correlation is exactly what people learn to break: they find the cheapest way to move the number that does not require doing the underlying work. The proxy and the goal, once loosely tied together, come apart — and the number keeps rising while the confidence it was supposed to represent falls.
BEFORE the metric is a target AFTER the metric is a target ───────────────────────────── ──────────────────────────── coverage ↑ because tests exercise coverage ↑ because people add more risky code the cheapest lines that count │ │ ▼ ▼ confidence ↑ too confidence FLAT (or ↓) (metric and goal move together) (metric and goal decouple)
the arrow from "number" to "goal" is what targeting snaps.Each metric in this Part, gamed
Section titled “Each metric in this Part, gamed”This is not abstract. Every measure we built in this Part has a well-worn gaming move, and each one satisfies the letter of the metric while betraying its spirit.
Coverage, padded with assertion-free tests
Section titled “Coverage, padded with assertion-free tests”A 90% coverage gate says “execute 90% of the lines.” It does not say “check that they are correct” — coverage counts execution, not assertion, a distinction Why 100% Is Not Bug-Free hammered. So the cheapest way to clear the gate is to write tests that run code and assert nothing:
def test_process_order(): process_order(sample_order()) # runs 40 lines. asserts NOTHING. # no assert — but every line it touched is now "covered"That test moves coverage exactly as much as a real test would, and catches exactly zero bugs. Point a gate at coverage and, under deadline pressure, this is what you manufacture: green badges sitting on top of tests that would not notice if the function returned garbage. The number went up; the confidence did not move.
Mutation score, inflated with trivial cases
Section titled “Mutation score, inflated with trivial cases”Mutation testing was supposed to be the cure for that — it grades whether a test would object when the code breaks. But it, too, is just a number, and any number can be gamed. If the target is “hit 80% mutation score,” you can raise the score without improving the hard code by piling tests onto the parts that are trivial to kill mutants in:
module mutants killed score ───────────────────────────────────────────── getters/setters 60 60 100% ← trivial, easy kills config parsing 20 18 90% PRICING ENGINE 40 14 35% ← the code that matters ───────────────────────────────────────────── whole module 120 92 77% → "add 5 getter tests" → 80% ✓The overall score crosses the target because you flooded it with easy kills, while the pricing engine — the one place a surviving mutant is genuinely dangerous — stays at 35%. The aggregate number now hides the risk it was invented to expose. A vanity mutation score is a real thing, and it is what “hit 80%” produces.
Escape rate, massaged in the tracker
Section titled “Escape rate, massaged in the tracker”Even the honest, lagging metrics are not safe. Escape rate is only as truthful as the field that feeds it — “where was this defect first found?” Make escape rate a number a team is punished for, and the reporting changes, not the reality: production bugs get quietly relabeled “staging findings,” logged as “enhancements,” or not filed at all. MTTR gets massaged the same way when timestamps are typed from memory during a post-mortem, drifting toward flattering numbers. You have not shipped fewer bugs; you have blinded the metric that was counting them.
The mechanism — optimizing the proxy instead of the goal
Section titled “The mechanism — optimizing the proxy instead of the goal”Step back and the pattern is identical every time. There is a goal you actually care about — justified confidence that the software works — and a proxy that is cheap to measure and usually tracks it. You cannot put a gate on the goal; “justified confidence” is not a number. So you put the gate on the proxy. And the instant there is a reward for the proxy, human ingenuity finds the shortest path to it:
GOAL (what you want) PROXY (what you measure) justified confidence ~≈ coverage %, mutation %, escape rate the code works numbers on a dashboard │ │ │ you can't gate the goal, │ so you gate the proxy… │ so you gate the proxy ───────┘ ▼ …and now people optimize the PROXY, because that's what's rewarded. The cheapest way to move the proxy is almost never the way that moves the goal. So the proxy climbs and the goal stands still.The failure is not that people are lazy or dishonest — it is structural. A gate creates a gradient, and effort flows downhill toward the cheapest way to satisfy it. If the cheapest way to hit 90% coverage is assertion-free tests, that is what a rational, time-pressed engineer will produce, whether or not they ever articulate it. The metric stops measuring test quality and starts measuring how hard people are being pushed to move the metric. That is Goodhart’s law operating exactly as written.
Guardrails — using metrics without being fooled by them
Section titled “Guardrails — using metrics without being fooled by them”Goodhart’s law is not an argument for throwing metrics away. Coverage, mutation score, and defect metrics are genuinely useful — as instruments. The law is a warning about one specific misuse: turning an instrument into a target. A few guardrails keep them honest.
- Use metrics as diagnostics, not verdicts. A metric’s job is to make you ask a better question.
"63% mutation score on the pricing module"should trigger “which mutants survived, and do they matter?” — a conversation — not “add tests until it says 80%.” The number is where the investigation starts, never where it ends. - Read the trend and the shape, not the single value. One number in isolation invites gaming; a trend across releases, and the distribution across modules, is far harder to fake and far more informative. A whole-repo mutation score of 77% hides the pricing engine at 35% — so read per-module, not the aggregate.
- Never make a metric an individual KPI. The fastest way to destroy a metric is to tie one person’s review or bonus to it. That is a reward attached to a proxy — Goodhart’s most potent trigger. Coverage on a dashboard is a team diagnostic; coverage on a performance review is a cobra farm.
- Correlate a gated metric against an ungated one. Keep at least one number nobody is optimizing (escape rate is ideal — it is lagging and hard to fake). If your gated proxy improves while the ungated one worsens, the gate is being gamed. This is the single most reliable Goodhart detector you have.
- Prefer soft gates and human judgment over hard pass/fail. “Coverage dropped 4 points in this PR — explain why” invites a real reason (it might be perfectly fine). “Merge blocked: coverage below 90%” invites a padding test. A gate that starts a conversation is far harder to game than one that just needs a number pushed over a line.
The part thesis, restated
Section titled “The part thesis, restated”Everything in this Part reduces to one sentence, and Goodhart’s law is why it is phrased so carefully:
Coverage, mutation score, escape rate — every metric here is a guide, never the goal.
The goal is, and has always been, justified confidence that the software works and a clear head about the edge cases that break it. The metrics are lenses that help you find where that confidence is thin. Point a lens at your suite and it shows you where to look. Turn that lens into a scoreboard and reward the score, and people will paint the scoreboard while the thing it was supposed to reflect quietly rots. Measure to learn; the moment you measure to judge a person, you have started measuring something else.
The architect’s lens
Section titled “The architect’s lens”- Why does it exist? Because a metric is a proxy, and every proxy can be optimized directly instead of the goal behind it. Goodhart’s law names the failure so you can recognise it before your green dashboard lulls you into shipping bugs.
- What problem does it solve? It explains, in one line, why gameable coverage, assertion-free tests, and vanity mutation scores keep appearing — they are not accidents or bad apples, they are the predictable result of putting a reward on a proxy.
- What are the trade-offs? Heeding it means giving up the comfort of a single hard pass/fail gate, in exchange for softer signals and human judgment that are harder to automate but far harder to fool. You trade a tidy number for an honest one.
- When should I avoid it? Never avoid the law — but do not weaponise it as an excuse to abandon measurement entirely. “Metrics can be gamed” is not a licence to fly blind; it is an instruction to use metrics as diagnostics rather than targets.
- What breaks if I remove it? You forget that the number is a stand-in, start trusting it literally, and gate on it. Coverage climbs, mutation score is gamed, escape rate is massaged, and everyone celebrates a dashboard that has decoupled entirely from whether the software works.
Check your understanding
Section titled “Check your understanding”- State Goodhart’s law and explain the causality in it — why does making a measure a target make it a worse measure, not just a useless one?
- Give the specific gaming move for coverage and for mutation score. For each, explain how it satisfies the letter of the metric while betraying its spirit.
- The page calls the failure “structural,” not a matter of lazy or dishonest people. What does “structural” mean here, and why does it matter for how you respond?
- You run coverage as a hard 90% merge gate. Name two guardrails from this page that would make that metric harder to game, and say what each one changes.
- Restate the thesis of this whole Part in one sentence, and explain what role metrics do play if they are not the goal.
Show answers
- When a measure becomes a target, it ceases to be a good measure. Targeting it makes it worse, not merely useless, because a reward on the proxy makes people optimize the cheapest way to move the number — which is almost never the way that moves the underlying goal. The proxy and the goal, once correlated, decouple: the number rises while the confidence it represented stays flat or falls, so the number is now actively misleading.
- Coverage: write tests that execute code but assert nothing — they move coverage exactly like a real test yet catch zero bugs, satisfying “run 90% of lines” while betraying the spirit (“the lines are checked”). Mutation score: pile tests onto trivial code (getters/setters) with easy kills to cross an aggregate target, while the hard, dangerous module stays weakly tested — satisfying “80% overall” while the aggregate now hides the risk it was meant to expose.
- “Structural” means the failure follows from the incentive, not from the people: a gate creates a gradient and effort flows toward the cheapest way to satisfy it, so any rational, time-pressed engineer produces the gaming behaviour whether or not they intend to. It matters because the fix is not “hire better people” or “tell them to try harder” — it is to change the structure: don’t put rewards or hard gates on proxies; use them as diagnostics instead.
- Any two of: read the trend and per-module distribution rather than the single aggregate (a 90% aggregate can hide a 35% critical module); correlate against an ungated metric like escape rate (if coverage rises while escapes rise, the gate is being gamed); make it a soft gate (“coverage dropped — explain why”) instead of a hard block, which invites a real reason rather than a padding test; and never tie it to an individual KPI, the most potent Goodhart trigger. Each replaces a single gameable number with a signal that is harder to fake or that starts a conversation.
- Every metric in this Part — coverage, mutation score, escape rate — is a guide, never the goal. The goal is justified confidence that the software works (and clarity about the edge cases that break it). Metrics are lenses/diagnostics that show you where that confidence is thin so you know where to look — useful precisely as long as you never mistake moving the number for reaching the goal.