Curing Flaky Tests

The previous page diagnosed why boundary tests flicker: shared state that leaks between tests, timing assumptions baked into sleep, uncontrolled clocks and randomness, order dependence, and the network. A diagnosis is not a cure. This page is the toolbox — the specific, boring disciplines that turn a suite you have learned to re-run into a suite you can believe on the first pass.

Hold on to one framing before any technique. A flaky test is a non-deterministic function: the same code and the same inputs sometimes return green and sometimes return red. Every cure below is really the same move — remove one source of non-determinism until the function is total. When nothing is left to vary, the test can only fail for the one reason you built it to catch. That is the whole game.

Cure 1: fresh state per test

The single largest source of flakiness is shared mutable state. Test A writes a row, forgets to clean it up, and Test B — which happened to run next today but not yesterday — sees a user that “shouldn’t” exist. The bug is not in either test’s logic; it is that neither test started from a known world. The cure is an ironclad rule: every test begins from state it fully controls, and no test can observe another’s leftovers.

There are three standard ways to get there, in rough order of speed:

   TRANSACTIONAL ROLLBACK        PER-TEST SCHEMA/DB          FRESH CONTAINER
   ┌──────────────────┐          ┌──────────────────┐        ┌──────────────────┐
   │ BEGIN            │          │ CREATE SCHEMA t7 │        │ docker run pg    │
   │  ...run test...  │          │  ...run test...  │        │  ...run test...  │
   │ ROLLBACK         │          │ DROP SCHEMA t7   │        │ docker rm -f     │
   └──────────────────┘          └──────────────────┘        └──────────────────┘
   fastest, same DB              medium, real isolation      slowest, total isolation
   (no DDL, no cross-conn)       (parallel-safe)             (fresh binary + fs)

Transactional rollback. Open a transaction in setup, run the test inside it, and roll it back in teardown. The database is untouched afterward, so the next test sees a pristine world. Fast, because nothing is actually committed. The catch: it only works when the code under test does not need its own separate connection or its own transaction, and it cannot test commit-time behaviour (triggers on commit, cross-transaction visibility).
Per-test schema or database. Give each test (or each worker) its own namespace — a fresh Postgres schema, a throwaway SQLite file, a uniquely named database. Slower than rollback but fully parallel-safe and able to test real commits.
Fresh container. Spin a real dependency in a disposable container (Testcontainers-style) per test class or suite. Highest fidelity and highest cost; reserve it for the seams that genuinely need a real engine.

When none of those fit — a shared external sandbox, say — fall back to unique fixtures: never hard-code user@example.com; generate user-{uuid}@example.com so two tests, or two runs, can never collide on the same key.

Isolation is not free, and the trade is worth measuring rather than guessing.

Suite: 800 integration tests, one shared Postgres

Strategy              Per-test setup   Total (serial)   Parallel-safe?
transactional rollback   ~1 ms           ~1–2 min          no*
per-test schema          ~15 ms          ~4–6 min          yes
fresh container          ~800 ms         ~11 min           yes

* rollback is single-connection safe but not safe to run
  many tests against one DB concurrently without more care.

Rollback is ~10× cheaper per test than a fresh schema and ~800× cheaper than a fresh container — but buys the least isolation. Most suites end up layered: rollback for the many small DB tests, containers for the few that must exercise a real engine end to end.

Cure 2: wait on a condition, never on a clock

The sleep(2000) in an async test is a bet: the thing I’m waiting for will finish within two seconds. On a fast laptop it wins; on a loaded CI runner it loses, and the test fails for a reason that has nothing to do with your code. Worse, to make it “reliable” people inflate the sleep — now every run pays two seconds whether or not it needed them, and the suite crawls. A fixed sleep is simultaneously too short (flaky under load) and too long (slow when idle). It is never the right number.

Replace the guess with an explicit wait: poll for the actual condition, or await an actual signal.

# FLAKY: a bet on wall-clock duration
create_order()
sleep(2)                         # "should be processed by now"... usually
assert order_status() == "SHIPPED"

# DETERMINISTIC: assert the condition, bounded by a generous timeout
create_order()
wait_until(lambda: order_status() == "SHIPPED", timeout=10, poll=0.05)
assert order_status() == "SHIPPED"

The difference is decisive. wait_until returns the instant the condition holds — so it is fast when the system is fast — and only fails if the condition never becomes true within a generous ceiling. The timeout is now a safety net for a genuine hang, not a load-bearing guess about duration. Prefer awaiting a real signal (a webhook, a message on a queue, a future that resolves) over polling when the system offers one; poll when it does not.

The rule: the timeout should be long enough that only a real bug ever hits it, and the poll should be short enough that a healthy system barely notices it. You are separating “how long am I willing to wait for a hang” from “how long the work actually takes” — a sleep fatally conflates the two.

Cure 3: control the nondeterminism

Some inputs vary every run by nature: the clock, the random-number generator, and the network. If a test consumes them raw, its result is a coin toss. The cure is to make each one an injected, controllable input rather than an ambient fact.

Freeze the clock

Any test touching “now” — expiry, TTLs, scheduling, “created today” — must not read the real system clock. Inject the clock as a dependency and hand the test a fake one you can set and advance.

// Instead of Instant::now() buried in the code, take a clock:
fn is_expired(token: &Token, clock: &impl Clock) -> bool {
    clock.now() >= token.expires_at
}

// The test owns time. No sleeping, no midnight-rollover flake.
let clock = FakeClock::at("2026-01-01T00:00:00Z");
let token = issue(&clock, ttl = Duration::hours(1));
clock.advance(Duration::minutes(61));
assert!(is_expired(&token, &clock));   // deterministic, and instant

This kills two flake classes at once: tests that fail at midnight or across a DST change, and slow tests that literally sleep to let a TTL elapse. Advancing a fake clock is free and exact.

Seed the randomness

A test that shuffles, samples, or generates IDs from an unseeded RNG produces a different world each run — and fails only on the unlucky draw. Seed it. A fixed seed makes the “random” sequence identical every time, so a passing test stays passing and a failing one is reproducible. (The exception is property-based testing, where you want varied inputs — but a good property framework prints the failing seed so you can replay it deterministically. Random inputs, reproducible failures.)

Pin the network with record/replay

Live third-party calls are the flakiest input of all: latency, rate limits, and outages you do not control. For tests whose subject is your logic rather than their uptime, record the real HTTP interaction once and replay it thereafter — the VCR pattern (named after the Ruby vcr gem; equivalents exist across ecosystems).

   First run (record):   test ──► real API ──► response saved to cassette.json
   Every run after:      test ──► cassette.json (no network at all)

Same request, same recorded response, every run — the network stops being an input. Two disciplines keep this honest: commit the cassettes so they are reviewable, and re-record on a schedule so a silently changed upstream contract is caught (this is exactly the drift that contract testing exists to catch on purpose). Record/replay removes flakiness from your suite; it does not remove your dependency on their contract holding.

Cure 4: detect and triage flakiness on purpose

You cannot fix what you cannot see, and flakiness hides — a test that fails 1-in-200 looks fine on the day you write it. Turn detection into a routine rather than a surprise.

Run tests in random order. Order dependence is invisible while the order is stable. Randomize the order (with the seed printed, per Cure 3) and hidden coupling surfaces immediately as a failure you can reproduce. A suite that only passes in one order is already broken; you just hadn’t run the order that proves it.
Retry to detect, not to hide. Run the suite (or a suspect test) many times — nightly --runs=50, or a stress job — and flag any test that is not 100% green. This is the opposite of a blind retry(3) in the run: one measures the flakiness rate so you can fix it; the other papers over it so you never see it.
Quarantine, don’t ignore. When a test is proven flaky and you can’t fix it this hour, move it to a quarantine lane: it still runs and still reports, but it does not fail the build or block the merge. Quarantine is a hospital bed with a discharge date, not a graveyard — a quarantined test carries a ticket and a deadline, or it will sit there forever eroding trust.
Track a flakiness rate as suite health. Treat ”% of test runs that fail on a passing commit” as a first-class metric, like coverage or build time. A rising number is a leading indicator that the suite is decaying toward the state where people re-run reflexively and stop reading red at all.

   DETECTED FLAKE  ──►  QUARANTINE (still runs, doesn't block)
                         │  file a bug, set a deadline
                         ▼
                        FIX the root cause (Cures 1–3)
                         │
                         ▼
                        RETURN to the blocking suite
                        (or DELETE if it earns nothing)

Cure 5: the cultural rule — a flake is a bug, not a retry

Every technique above fails if the culture around the suite is wrong, so state the rule plainly: a flaky test is a bug with a ticket, not a retry(3) to paper over.

The temptation is overwhelming. A test fails once in CI, a re-run goes green, and it is so easy to wrap it in an automatic retry and move on. But a blind auto-retry does not fix anything — it hides something. And the thing it most often hides is a real race condition in your product, not your test: if the code only works when the timing happens to line up, users on a slow network will hit exactly the failure your retry just swept under the rug. The flaky test was a true alarm. Auto-retry silenced the alarm and left the fire.

The discipline that keeps a suite trustworthy:

First flake is a bug report. File it, tag it, and treat it like any other defect — because it is one, and the defect may be in the product.
Retry to detect, never to hide. Stress-run to measure flakiness (Cure 4); never add a silent in-line retry to make red go green.
Green must mean green. The whole value of a suite is that a pass is trustworthy. Every papered-over flake erodes that until people re-run reflexively — at which point the suite costs time and buys no confidence, the worst possible trade.

On a summer afternoon in 2012, a US brokerage firm deployed new trading software to its production servers. Because of a deployment that left old and new code paths live on different machines at once, a dormant flag re-activated retired logic, and the system began firing millions of unintended orders into the market within minutes. The loss was around US$440 million — roughly the firm’s entire capital — and it effectively ended the company in days.

The failure was a state-and-timing problem across a boundary: which code was live on which server, and in what order things ran. That is precisely the class of bug a flaky test is often pointing at. A test that fails intermittently under real timing is not noise to retry away — it can be the only warning you get that a race exists before it reaches production. Treat the intermittent red as the signal it is.

Putting it together

The cures compose into a single pipeline for any test you don’t trust:

   1. ISOLATE      fresh state per test  ── kills shared-state leaks
   2. WAIT         condition, not clock  ── kills timing guesses
   3. CONTROL      fake clock, seed RNG,  ── kills ambient inputs
                   record/replay net
   4. DETECT       random order + stress  ── surfaces what's left
                   + quarantine + metric
   5. CULTURE      flake = bug, not retry ── keeps green meaning green

Work top-down. Most flakiness dies at steps 1-3 because those remove the sources of non-determinism outright. Steps 4-5 exist for the residue and for the humans: they surface what slipped through and enforce that a flake is fixed rather than hidden. Do all five and a passing suite becomes a fact you can act on — which is the entire reason the suite exists.

The architect’s lens

Why does it exist? Because integration and E2E tests touch real state, real time, and real networks — the very things that vary between runs — so without deliberate discipline they become non-deterministic and stop buying confidence.
What problem does it solve? It converts a test suite from something people re-run and second-guess into something whose green is trustworthy on the first pass, protecting the confidence the tests were written to create.
What are the trade-offs? Determinism costs setup and speed: per-test isolation and controllable clocks/RNG/network are more infrastructure than “just call the real thing,” and record/replay can drift from reality if you never re-record.
When should I avoid it? Never avoid the goal of determinism — but avoid the heaviest tools where they don’t fit: don’t spin a fresh container when a rollback suffices, and don’t stub the network when the thing you’re actually testing is the live integration.
What breaks if I remove it? Flakiness compounds, engineers re-run reflexively, red gets ignored, and eventually a real race — a Knight-Capital-shaped timing bug — sails through a suite everyone has learned to distrust.

Check your understanding

A test passes alone but fails when the full suite runs. Which cure is most likely missing, and what are two concrete ways to implement it?
Why is a fixed sleep(2) “simultaneously too short and too long,” and what replaces it?
Name the three ambient inputs that make a test non-deterministic and the cure for each.
Distinguish “retry to detect” from “retry to hide.” Which one belongs in your CI run, and why?
State the cultural rule in one sentence and explain what real-world danger a blind auto-retry can conceal.

Show answers

Isolation / fresh state per test is missing — the test depends on state a sibling created or left behind. Implement it with transactional rollback (begin in setup, roll back in teardown), a per-test schema/database (each test gets its own namespace, parallel-safe), a fresh container per suite, or at minimum unique fixtures (user-{uuid}@…) so keys can’t collide.
It is too short because a loaded CI runner may not finish the work in two seconds (flaky under load), and too long because when the system is fast you still pay the full two seconds (slow when idle) — it is never the right number. Replace it with an explicit wait: poll for the actual condition (or await a real signal) up to a generous timeout that only a genuine hang would ever hit.
Time — inject a fake clock you can set and advance instead of reading the system clock. Randomness — seed the RNG so the sequence is identical (and reproducible) every run. Network — record the real interaction once and replay it (VCR pattern) so the same request yields the same response with no live call.
Retry to detect runs a test many times deliberately (e.g. nightly --runs=50) to measure its flakiness so you can fix the root cause. Retry to hide is a blind in-line retry(3) that reruns on failure until green, masking the problem. Only detect belongs — an in-line retry in the run destroys the meaning of a pass and can conceal a real product race.
A flaky test is a bug with a ticket, not a retry(3) to paper over. A blind auto-retry can conceal a genuine race condition in the product (not the test): if the code only works when timing lines up, a silenced flake means real users on slow paths hit the failure the retry swept away — the alarm was true; the retry muted it.