Skip to content
Software Testing · First Principles

Why 100% Coverage Is Not Bug-Free

The previous page split coverage into line, branch, condition, and path — four increasingly demanding ways to ask “did this code run?” Each is stricter than the last, but notice they are all variations on the same verb: run. None of them upgrades to check. This page confronts the number those metrics roll up into, the most dangerous number in testing: a green 100%.

100% coverage feels like a finish line. It is printed on a badge, gated in CI, and celebrated in standups. But every coverage metric answers exactly one question — was this line executed by some test? — and that question is not the one you care about. You care whether the code is correct. This page shows, with concrete bug classes, why those are different questions, and why a suite can hit 100% while catching nothing that matters.

The claim is not “coverage is bad.” Low coverage is a real, useful alarm, and we will keep saying so. The claim is narrower and sharper: 100% is a floor you have finished walking, not a proof the building is sound. The number saturates — there is nothing above 100% — long before the interesting testing questions are answered, and that saturation is exactly what makes it dangerous. A metric that tops out early lulls you into stopping early.

Coverage measures execution, not correctness

Section titled “Coverage measures execution, not correctness”

Recall what coverage actually measures: the instrumentation records which lines the interpreter touched while your tests ran. That is all. A line is “covered” the instant control flow passes through it — regardless of what the test then does, or fails to do, with the result.

Correctness lives in two places coverage never looks:

   what coverage sees            what coverage cannot see
   ──────────────────            ────────────────────────
   this line executed      ✓     did any test CHECK the result?   ✗
                                 was the result checked against
                                 the RIGHT expected value?        ✗
                                 was this line fed the INPUT
                                 that makes it wrong?             ✗

Coverage is a reachability fact. Correctness is a behavioral fact. A line can be perfectly reachable and completely wrong. The rest of this page walks three bug classes that prove it — each survives a 100% report unharmed.

The trap is that “reachable” sounds like “tested.” In ordinary speech, to reach a place is to have been there and looked around. But an instrumented line has no eyes. When control flow walks past a counter, the counter learns exactly one bit — arrived — and nothing about what happened on arrival. A test that reaches a line and asserts its output, and a test that reaches the same line and asserts nothing, leave identical footprints in the coverage table. The tool cannot tell a careful visitor from a tourist. That single blind spot is the source of every failure on this page.

The two spaces that never line up

Section titled “The two spaces that never line up”

The confusion has a precise geometric cause. Every test lives in two different spaces at once, and coverage only measures one of them.

   CODE SPACE (finite)              INPUT SPACE (effectively infinite)
   ──────────────────              ──────────────────────────────────
   line 1                          age = ... -3, -1, 0, 17, 18, 19, ...
   line 2   ← coverage counts      list = [], [x], [x,y], [x,y,z], ...
   line 3     how many of these    string = "", "a", "  ", 10^6 chars ...
   ...        your tests reached

Coverage lives entirely in code space — it counts how many of a finite, countable set of lines your tests touched. Bugs live in input space — the effectively infinite set of values a line can be handed. You can reach 100% of code space while sampling a vanishingly thin slice of input space. A metric confined to the small, finite space simply cannot speak to correctness, which is a property of the large, infinite one. Every bug class below is one instance of this mismatch: full code-space coverage, near-zero input-space coverage.

Put numbers on the mismatch and it stops being abstract. A one-line function return age > 18 is one line — a code space of size one, which a single test drives to 100%. But age is (say) a 32-bit integer: over four billion possible values, the input space that actually decides whether the line is correct. One test covers 100% of the code and roughly 1 / 4,000,000,000 of the inputs. The coverage report will not hint at the ratio. It reports “100%” and falls silent about the four billion values you did not try — including, as we are about to see, the one at the boundary that is wrong.

Bug class 1 — the missing assertion

Section titled “Bug class 1 — the missing assertion”

The purest way to get 100% coverage while testing nothing is to run the code and never check what it produced. Coverage counts the execution; it has no idea whether an assert followed.

def apply_discount(price, percent):
    return price - (price * percent / 100)   # bug: no clamp, percent>100 → negative


def test_apply_discount():
    result = apply_discount(100, 20)   # line executes → 100% covered
    # ...and that's it. No assertion. Nothing is checked.

The function line runs. Coverage reports it covered. The test passes — because a test with no failing assertion always passes. You could replace the function body with return 0, or return "banana", and this test would still be green. The coverage number is identical whether the code is right or catastrophically wrong.

This is not a strawman. Missing assertions creep in through refactors (“I’ll add the check later”), through tests that only assert no exception was thrown, and through the exact failure mutation testing is designed to catch — which is why the next page exists. The lesson is stark: coverage counts the visit, not the verdict.

There is an even quieter version of this bug: the weak assertion. A test that writes assert result is not None or assert len(output) > 0 does technically check something — so it looks disciplined — while constraining almost nothing about the value. The line runs, coverage is 100%, an assertion exists, and yet nearly every wrong output still slips through because the assertion is loose enough to accept it. Coverage cannot grade an assertion’s strength any more than it can notice one’s absence. Both leave the same footprint: a covered line and a passing test. Reaching for coverage to catch this is reaching for the wrong instrument entirely.

Bug class 2 — the wrong boundary on a covered line

Section titled “Bug class 2 — the wrong boundary on a covered line”

Now add an assertion, so the test genuinely checks something. Coverage still cannot tell you that you checked it at the right input. Watch a classic off-by-one survive a full report.

def is_eligible(age):
    return age > 18          # bug: should be >= 18; 18-year-olds are eligible


def test_is_eligible():
    assert is_eligible(25) == True    # line covered, assertion passes
    assert is_eligible(10) == False   # line covered, assertion passes

Every line of is_eligible is executed. Both assertions pass. Coverage reports 100%. And the function is wrong: an 18-year-old is rejected. The bug lives at the boundary — age == 18 — and neither test supplied that value. The covered line does the wrong thing for exactly the one input no test tried.

This is the deep point of the empty, null & boundaries mindset colliding with coverage: off-by-one and boundary bugs routinely survive 100% coverage, because the metric rewards executing the line, not executing it at the value where the operator matters. > and >= compile to different code but produce identical coverage for the inputs 25 and 10. Only a test at 18 can tell them apart, and coverage will never ask you to write it.

   is_eligible(age):  return age > 18


   input   >  18   should be >= 18   test supplied it?
   ─────   ──────  ───────────────   ────────────────
    25     True         True (ok)          yes ✓
    10     False        False (ok)         yes ✓
    18     False   ←     True   ← BUG       NO  ✗   ← 100% coverage, bug intact

Bug class 3 — untested input values on a covered line

Section titled “Bug class 3 — untested input values on a covered line”

The third class generalizes the second. A single line can be correct for the inputs you tried and wrong for a whole category you never tried — the infinite input space leaking through a covered line.

def average(numbers):
    return sum(numbers) / len(numbers)   # crashes on []  (ZeroDivisionError)


def test_average():
    assert average([2, 4, 6]) == 4   # line covered, passes, 100%

The line is covered. The assertion is real and passes. Coverage is 100%. But feed it the empty list — a value the test never supplied — and it throws ZeroDivisionError. This is the null / empty-input bug that “full coverage” happily hides: the same executed line is fine for [2, 4, 6] and broken for []. Coverage said nothing about inputs, because coverage cannot see inputs — it sees only whether control flow arrived.

Restate the principle from all three classes: a covered line can be wrong for values the test never supplied. Coverage is a property of the test run, not of the input space. Two suites can produce byte-identical 100% reports while one probes the boundaries and empties and the other never leaves the happy path.

Why the three classes are really one bug

Section titled “Why the three classes are really one bug”

Look at what the three bug classes have in common and they collapse into a single shape. In every case the line executed, so coverage was satisfied; and in every case something about the input-or-oracle pairing was never exercised, so the bug survived:

   bug class              line ran?   what was never exercised
   ────────────────────   ─────────   ───────────────────────────────
   1  missing assertion      yes      the OUTPUT — no oracle at all
   2  wrong boundary         yes      the boundary INPUT (age == 18)
   3  empty / null input     yes      the empty INPUT ([])

The column that decides correctness — “what was never exercised” — is the column coverage cannot see. This is why chasing a higher coverage number is the wrong reflex when a bug escapes: the number was already green. What was missing was never a line; it was an (input, expected-output) pair. Coverage counts lines. Bugs hide in pairs. The two are measured in different currencies, and no amount of one buys the other.

The false-confidence failure mode

Section titled “The false-confidence failure mode”

The danger is not that coverage is useless — low coverage is a genuine, actionable signal that whole regions of code have never run under test. The danger is what a high number does to human behavior.

A 100% badge is an authority. It ends arguments. “We’re at 100%, ship it.” “Do we need a test for the empty case?” — “Coverage is already green.” The number becomes a stopping rule, and it stops teams at precisely the moment the interesting work begins: choosing which boundary, null, and adversarial inputs to bet on. The metric that was meant to reveal untested code ends up concealing untested behavior behind a reassuring green.

Worse, the failure is invisible from inside. A team at genuine 60% coverage knows it has gaps and stays alert. A team at 100% has been handed a certificate of completeness by a tool that never checked completeness of anything but execution. False confidence is more dangerous than acknowledged ignorance, because ignorance keeps you looking and false confidence tells you to stop.

This is a preview of Goodhart’s Law: the moment “100% coverage” becomes the target, it stops measuring test quality. You can always reach 100% by deleting assertions or padding tests that execute lines without checking them. The number goes up; the confidence it implies goes down. A team optimizing the badge and a team optimizing for caught bugs will write very different suites — and only one of them survives contact with production.

The fix is not to abandon coverage — it is to demote it from verdict to prompt. A zero-hit line is a genuine finding: go write a test. But a green line is a question, not an answer. The useful discipline is to read every covered line and ask three things the number cannot:

   for each covered line, ask:
   1. does a test ASSERT this line's result?          (kills bug class 1)
   2. is the assertion checked at the BOUNDARY?        (kills bug class 2)
   3. was the line fed the EMPTY / NULL / extreme?     (kills bug class 3)

Coverage can tell you the line ran. Only you — or a tool that reasons about assertions, like mutation testing — can answer those three. That is the entire gap between “100%” and “bug-free,” written as a checklist: the number closes the first column and leaves the other three entirely to you.

The uncomfortable takeaway from that table is that the badge and the danger scale together. The more a team leans on the number, the more incentive there is to reach it the cheap way — run lines, skip assertions, avoid the awkward inputs — and the cheaper the 100% becomes. A metric that gets easier to satisfy the harder you push on it is not a safety net; it is a mirror that tells you what you already decided to believe.

Under the hood — reframing what 100% actually asserts

Section titled “Under the hood — reframing what 100% actually asserts”

The cleanest cure is to read the number literally. “100% line coverage” does not mean “every behavior is correct.” It means, precisely:

“Every line ran at least once, on at least one input, and nothing about the result was necessarily checked.”

That is a much smaller claim than the badge implies. Line coverage is really “every line ran once.” Branch coverage is “every branch ran once.” Neither says a word about (a) whether the output was asserted, (b) whether it was asserted against the correct oracle, or (c) which of the astronomically many inputs the line was run on. Coverage answers “can a test reach this code?” — a reachability question. Correctness needs “does this code produce the right answer for the inputs that matter?” — a behavioral question no coverage tool can pose.

If you read the number that literally every time you see it, the false confidence evaporates on contact. “100%” stops meaning “we are done” and starts meaning “every line ran at least once — now the real testing begins.” That reframing is the single most valuable habit this page can leave you with.

Hold both facts at once: coverage is a useful floor (it finds code that no test touches at all) and a useless ceiling (it cannot certify that touched code is right). Chase it up from 40% and you find dead, untested regions. Treat the 100% at the top as a proof of correctness and you have simply relocated your bugs from “known untested” to “believed tested.”

Even the strongest structural metric cannot close the gap. Recall from coverage types that path coverage is the most demanding of all — it wants every route through the branching structure exercised. Suppose you somehow reached 100% path coverage. You would have exercised every route, but each route still runs on whatever inputs the test happened to supply, and there are effectively infinite inputs per route. average(numbers) has a single path — no branches at all — yet is correct for [2, 4, 6] and broken for []. The most stringent coverage metric in existence, at a perfect 100%, would still not force a test for the empty list. Structural coverage measures the shape of the code; correctness depends on the values flowing through that shape, and no structural metric can reach the values.

Every one of these lessons rhymes with the book’s throughline: justified confidence comes from tests that would fail on wrong behavior, and coverage is silent about whether a single test in your suite could ever fail. A 100% badge answers “did the code run?” The question that earns confidence is “would we have noticed if it were wrong?” — and that question has to be asked input by input, boundary by boundary, assertion by assertion. Coverage will not ask it for you; the next two pages, on mutation testing and defect metrics, take up the tools that will.

The architect’s lens

Section titled “The architect’s lens”

“100% coverage is not bug-free” is less a technique than a lens you apply to every coverage number you will ever see — so it is worth pinning down at altitude why the idea has to exist and what depends on it.

  • Why does the “100% ≠ bug-free” idea exist? Because coverage tools measure execution, a reachability fact, while people read the resulting number as correctness, a behavioral fact. The idea exists to keep those two apart before the confusion ships a bug.
  • What problem does it solve? It defuses the single most common false signal in testing — a green 100% badge — by naming exactly what it does and does not guarantee, so teams stop treating a coverage number as a proof of correctness.
  • What are the trade-offs? Taking it seriously means coverage can no longer be your finish line, so you need a second signal for test quality — assertion discipline and mutation testing — which costs effort the badge let you skip.
  • When should I avoid this framing? Never avoid the framing, but do keep using coverage as a floor: low coverage still reliably flags code no test touches. Distrust the ceiling, not the metric.
  • What breaks if I ignore it? Teams gate CI on 100% and feel safe, then off-by-one, null, and empty-input bugs sail through into production — because the number they trusted measured whether lines ran, never whether the code was right.

Check your understanding

Section titled “Check your understanding”
  1. In one sentence, what question does line coverage answer, and why is it not the question you actually care about?
  2. Write (in words) a test that achieves 100% coverage of a function while verifying nothing about its output. What is missing?
  3. is_eligible returns age > 18 but should return age >= 18. Both tests pass and coverage is 100%. Exactly which input reveals the bug, and why did coverage never demand it?
  4. Explain the “false-confidence failure mode”: how can a 100% badge cause a team to write fewer of the tests that matter?
  5. Restate what “100% line coverage” literally guarantees — and name two things it explicitly does not guarantee.
Show answers
  1. Line coverage answers “was this line executed by at least one test?” — a reachability fact. You actually care whether the code produces the correct result, a behavioral fact coverage never inspects.
  2. Call the function and simply don’t assert on the return value (e.g. result = apply_discount(100, 20) with no assert). The line executes, so coverage counts it, and a test with no failing assertion always passes. What’s missing is any check of the output — the verdict, not the visit.
  3. The input age == 18. The two tests supplied 25 and 10, which give the same answer under both > and >=, so they execute the line and pass. Coverage rewards executing the line, not executing it at the boundary value where > and >= diverge, so it never demanded a test at 18.
  4. A 100% badge acts as a stopping rule and an authority: “we’re green, ship it.” It ends the discussion right when the valuable work — choosing boundary, null, and adversarial inputs — should begin, so the metric meant to reveal untested code instead conceals untested behavior behind a reassuring number.
  5. It guarantees only that every line ran at least once on at least one input. It does not guarantee that any output was asserted (or asserted against the correct expected value), and it does not guarantee the line was run on the inputs that break it (boundaries, empty, null). “Every line ran once” is not “every behavior is correct.”