Debugging a Failing Test, Methodically

The last page taught you how to make a test deterministic — how to control time, randomness, and order so that a green test means “the code works” and a red test means “something is wrong.” This page is about what you do the moment a test does go red.

A red test is data, not a verdict. It is not proof the code is broken, and it is not proof your test is broken either. It is a single signal that says: the observed behaviour did not match the expected behaviour. Your job is to turn that one bit of information into a diagnosis. Beginners treat a failing test as an emergency and start changing things at random until the bar goes green. That is guessing, and it wastes hours. What follows is a method that replaces guessing with a short, repeatable loop.

The first question: is the test wrong, or is the code wrong?

Before you touch anything, answer one question: which side of the assertion is lying?

A test compares two things — what the code did and what the test expected. When they disagree, exactly one of the following is true:

• The code is wrong. It computed the wrong answer. The test caught a real bug — this is the test doing its job.
• The test is wrong. The code is correct, but the test’s expectation is stale, its setup is broken, or it asserts something the spec never promised. This is a false alarm, and it is just as common.

Confusing these two is the single biggest time-sink in debugging. If you assume the code is broken when the test is stale, you “fix” working code and make it worse. If you assume the test is stale when the code is broken, you weaken the test and ship the bug. So the very first move is to hold the question open:

   assertion failed
         │
         ▼
   ┌────────────────────────────┐
   │  is the EXPECTED value     │
   │  actually correct?         │
   └────────────────────────────┘
      │                    │
   yes│                    │no
      ▼                    ▼
  code is wrong        test is wrong
  (real bug)           (stale / bad expectation)

You answer it by going back to the source of truth — the specification, the ticket, the API contract, the product owner — not by staring at the code. Ask: “what is the correct behaviour here, independent of what either the code or the test currently says?” Only once you know the right answer can you tell which side drifted.

Both outcomes are legitimate results of a debugging session. “The test was wrong” is not a failure on your part; it is a finding, and a valuable one.

Make it reproducible before you investigate

You cannot debug what you cannot reproduce. An intermittent failure — green on this run, red on the next, with no code change in between — is the hardest kind, and every minute spent “investigating” a bug you can’t summon on demand is guesswork.

So the second move, before any hypothesis, is to make the failure happen on demand. Concretely:

• Run just the failing test in isolation. Most runners let you target one test: cargo test tcp_set_get_del_roundtrip, pytest path::test_name, go test -run TestName. If it fails alone, you have a clean, fast reproduction.
• If it only fails in the full suite, suspect shared state or order. A test that passes alone but fails in the suite is almost never about that test — it is about another test leaking state (a shared file, a global, a database row) or about test order. Re-run with a fixed seed and a pinned order to confirm (see Determinism).
• Loop it to expose flakiness. Run the test 100 times: for i in $(seq 100); do cargo test the_test || break; done. If it fails 3 times in 100, you have a flaky test — a nondeterminism bug that must be pinned down before the “real” investigation, because you can’t trust any experiment while the outcome is random.

A failure you can reproduce in one command, every time, is a failure you can debug. A failure you can’t reproduce is a research project. Turn the second into the first before you go further.

By the numbers — the cost of a flaky reproduction

Suppose a test fails intermittently 1 time in 10, and each run takes 20 seconds. You want to test a hypothesis by making a change and confirming the failure disappears.

Confidence that "it's fixed" after N green runs,
if the bug is really still there (10% failure rate):

  after 1 green run :  90% chance you got lucky   → basically no signal
  after 5 green runs:  0.9^5  = 59% still fooled
  after 20 green runs: 0.9^20 = 12% still fooled
  after 44 green runs: 0.9^44 =  1% still fooled

A 1-in-10 flake needs ~44 runs (≈15 min) per experiment
just to be 99% sure a change helped.

Now do that for five hypotheses and you have burned an afternoon. This is exactly why you first make the failure deterministic — ideally 100%-reproducible — so a single run gives a real answer, not a coin flip.

Localize: shrink the failing case

Once the failure is reproducible, you rarely know where it comes from. Localization is the art of shrinking the search space until the bug has nowhere left to hide. Two complementary tools do most of the work: minimization (shrink the input) and bisection (shrink the history).

Minimization — the smallest failing case

Take the failing scenario and remove everything that isn’t required to trigger it. Delete inputs, drop setup steps, cut fields from the payload — after each cut, re-run. If it still fails, the thing you removed was irrelevant; keep it removed. If it now passes, the last thing you removed was load-bearing; put it back.

You are converging on the minimal reproduction: the smallest input and shortest sequence of steps that still turns the test red. A three-line reproduction points at the bug; a 200-line integration scenario hides it. This is the same discipline a good bug report needs (see Writing a Bug Report That Gets Fixed).

Bisection — the commit that introduced it

If a test passed last week and fails today, the bug lives in a specific commit. You do not have to read every commit — you binary-search them. git bisect automates this: you mark one known-good commit and one known-bad commit, and git checks out the midpoint and asks you to test it. Each answer halves the range.

git bisect start
git bisect bad                 # current HEAD is broken
git bisect good v1.4.0         # this old tag was fine

# git checks out the midpoint; you run the test and report:
git bisect good     # or: git bisect bad

# repeat ~log2(N) times; git names the first bad commit
git bisect reset               # when done, return to HEAD

Across 1,000 commits, bisection finds the culprit in about 10 steps (log₂1000 ≈ 10), not 1,000. If the test is scriptable, git bisect run ./run-the-test.sh does even the reporting automatically — git drives the whole search while you get coffee. The commit it names tells you what change broke the behaviour, which is often 90% of the diagnosis.

Work by hypothesis, change one thing

Now you have a reproducible, minimal, localized failure. Resist the urge to start editing. Debugging is not editing; it is experimentation. The loop is:

1. Read the actual-vs-expected output. The assertion already told you a lot. expected 42, got 41 is an off-by-one. expected "2026-07-09", got "2026-07-08" is a timezone or clock bug. expected 200, got 500 plus the stack trace is a crash, not a wrong answer. Read it literally before theorizing.
2. Form one testable hypothesis. “The count is off by one because the loop uses <= instead of <.” A hypothesis is a specific, falsifiable claim — not “something’s wrong with the counting.”
3. Change one thing to test that hypothesis — add a log line, tweak one value, isolate one branch.
4. Observe. Did the outcome change the way your hypothesis predicted? If yes, you’ve confirmed it. If no, the hypothesis was wrong — revert the change and form the next one.

The failure mode this avoids has a name: shotgun debugging — changing five things at once, seeing green, and having no idea which change mattered (or whether you introduced two new bugs that cancel out). When you change one thing at a time, every run is a controlled experiment that teaches you something. When you change five, a green bar teaches you nothing.

   ┌─────────────┐
   │  read the   │
   │  assertion  │◄──────────────┐
   └─────────────┘               │
         │                       │
         ▼                       │ hypothesis wrong:
   ┌─────────────┐               │ REVERT, try next
   │ form ONE    │               │
   │ hypothesis  │               │
   └─────────────┘               │
         │                       │
         ▼                       │
   ┌─────────────┐     ┌─────────┴──────┐
   │ change ONE  │────►│  observe:      │
   │ thing       │     │  as predicted? │
   └─────────────┘     └────────────────┘
                              │ yes
                              ▼
                          diagnosed

Read the signal the test gives you

A failing test is a witness, and how much it tells you depends entirely on how well it was written. This is where the quality of the test pays off — or costs you.

Compare two failures of the same bug:

# Poor signal — sends you guessing:
assertion failed: result == expected

# Good signal — this is a diagnosis, not a mystery:
assertion `left == right` failed: cart total after 10% discount
  left:  90.00      (actual — code produced this)
  right: 89.99      (expected — spec says this)

The second one is nearly self-solving: you can see it’s a rounding difference of one cent on a discount, so you go straight to the rounding code. The first tells you only that something is wrong. Good assertion messages, structured logs around the failure, and readable diffs (of objects, JSON, images) turn a failure into a diagnosis; poor ones turn it into a scavenger hunt.

When you write the assertion, spend the extra line to say what was being checked and to print both values — your future self debugging at 5pm will thank you. When you’re debugging someone else’s poor assertion, your first move is often to improve the assertion or add a log, so the test tells you more on the next run. That’s not a detour; it’s building the diagnostic instrument you needed.

It actually happened — Knight Capital, 2012

On 1 August 2012, the trading firm Knight Capital deployed new code to its servers. A deployment step missed one of eight servers, and — per public accounts — an old, repurposed feature flag activated dormant code on that single machine. Within roughly 45 minutes the errant server sent millions of unintended orders into the market; the firm lost on the order of $440 million and nearly collapsed.

The debugging lesson is brutal and precise: the failure was intermittent by server — seven machines behaved, one did not. A bug that only reproduces on one host out of eight is exactly the kind that eludes anyone who doesn’t first pin down where it reproduces. Reproduce, then localize — before the blast radius grows. (Details are reconstructed from public accounts; treat specifics as approximate, the lesson as exact.)

Know when to escalate — and hand off cleanly

Sometimes you get to the “is it the test or the code?” fork and you genuinely can’t answer it yourself, because the answer depends on intended behaviour you don’t own. Is POST /shorten on a duplicate URL supposed to return the existing code or mint a new one? That’s a product decision. Guessing wastes your time and risks weakening a correct test or hiding a real bug.

That is the moment to escalate — not as surrender, but as the correct next step. The skill is handing off well, so the developer or product owner spends thirty seconds, not thirty minutes, catching up:

• The exact reproduction: one command, the minimal input, the pinned seed/order — everything from your localization work.
• Actual vs expected, quoted verbatim from the assertion output.
• The precise question, phrased as a decision: “Should duplicate URLs return the existing code (test’s current expectation) or a new one (code’s current behaviour)? The spec doesn’t say.”
• What you already ruled out, so they don’t repeat your steps.

A clean reproduction is the difference between “I think something’s flaky, can you look?” (which gets ignored) and “here’s the one command, here’s the one question” (which gets answered). More on this collaboration in Working With Developers.

The architect’s lens

• Why does it exist? Because a red test is ambiguous by nature — it says behaviour ≠ expectation but not which side is wrong or why. A method exists to resolve that ambiguity systematically instead of by luck.
• What problem does it solve? It replaces shotgun guessing with a bounded, repeatable loop — reproduce, localize, hypothesize, observe — so debugging time is spent gathering evidence, not flailing.
• What are the trade-offs? The method has up-front cost: isolating a reproduction and running git bisect feel slower than “just trying a fix.” That discipline pays off on every failure but the most trivial, and trivial failures don’t need a method anyway.
• When should I avoid it? When the assertion output already is the diagnosis — an obvious typo, a one-line stale expectation — fix it and move on. Don’t ceremonially bisect a bug you can already see.
• What breaks if I remove it? Debugging degrades into guessing: intermittent bugs never get pinned down, “fixes” that don’t fix anything get committed, and correct code gets “repaired” to satisfy a stale test. Time-to-diagnosis becomes unbounded and unpredictable.

Check your understanding

1. A test fails. Before changing any code, what is the very first question you should answer, and why does confusing the two possible answers cost so much time?
2. A test passes when you run it alone but fails inside the full suite. What does that pattern most strongly suggest, and how do you confirm it?
3. Explain how git bisect finds the offending commit in about 10 steps across 1,000 commits instead of checking all 1,000.
4. What is “shotgun debugging,” and which single rule in the hypothesis loop prevents it?
5. You reach the “is it the test or the code?” fork and can’t answer it because it depends on intended behaviour you don’t own. What do you do, and what four things do you include in the hand-off?

Show answers

1. Is the test wrong or is the code wrong? They are both real, common outcomes. If you assume the code is broken when the test’s expectation is merely stale, you “fix” working code and make it worse; if you assume the test is stale when the code is genuinely broken, you weaken the test and ship the bug. You answer it against the source of truth (spec/ticket/contract), not by staring at the code.
2. It strongly suggests shared state leaking between tests, or an order dependency — another test is mutating a file, global, or database row the failing test relies on, or the tests only fail in a particular order. Confirm by re-running with a pinned order and fixed seed; if the failure tracks order/seed rather than the test itself, that’s the cause.
3. Bisection is a binary search over history: mark one known-good and one known-bad commit, test the midpoint, and each answer (good/bad) halves the remaining range. log₂(1000) ≈ 10, so about 10 tests locate the first bad commit instead of 1,000.
4. Shotgun debugging is changing several things at once, seeing green, and having no idea which change mattered (or whether new bugs cancelled out). The rule that prevents it: change exactly one thing per experiment, then observe — and revert it if the hypothesis was wrong.
5. Escalate to the developer or product owner, because the fork depends on intended behaviour you don’t own and guessing risks either weakening a correct test or hiding a real bug. Hand off with: (1) the exact one-command reproduction with minimal input and pinned seed/order, (2) actual-vs-expected quoted verbatim, (3) the precise question phrased as a decision, and (4) what you already ruled out.