Flaky Tests: Why They Break

The previous page, End-to-End Tests and Their Cost, left us holding the most valuable and most fragile tests in the suite. Fragile is the word to sit with. As you climbed the pyramid — more real components, more network, more real time — you traded away the one property that made unit tests trustworthy: determinism. A unit test over a pure function either passes or fails, and it says the same thing every run. Bolt in a real database, a real socket, and a real clock, and suddenly the same code can pass at 2:14 and fail at 2:15.

That is a flaky test, and this page is a field guide to why it happens. We are not curing anything yet — the next page does that. The job here is diagnosis: to learn the signatures of the four great families of flakiness so that when a build goes red for no reason, you can name the cause instead of hitting “Re-run” and hoping.

This page sits at the far corner of the fidelity/speed/stability triangle the Part overview drew. Flakiness is what “stability” looks like when it fails: a test that a healthy system does not reliably pass. And it is not an accident of sloppy code — it is the predictable tax on the very things that made the earlier pages’ tests valuable. Real dependencies bring real network and real time. End-to-end tests bring the whole moving system into scope. Every source of fidelity is also a potential source of nondeterminism. So diagnosis begins by accepting that flakiness is where the fidelity is — and then narrowing to which source, in this test, is the culprit.

What flakiness actually is

A flaky test is one that passes and fails on the same code, with no change to the test or the system under test — only to the invisible conditions around it.

The key phrase is same code. A test that fails because you broke the code is not flaky; it is doing its job. A flaky test fails for reasons the test never meant to assert on: which order it ran in, what time it is, whether another test left a row in the database, whether a DNS lookup took 40 ms or 4 seconds. The system under test may be perfectly correct. The test is nondeterministic.

Why a flaky test is worse than no test

This is the counter-intuitive claim that governs the whole page, so it is worth proving. A test exists to buy justified confidence — the book’s throughline. Its value comes entirely from the meaning of its two outcomes:

Green means the behaviour is correct.
Red means the behaviour is broken — go fix it.

A flaky test destroys the second half of that contract. Red no longer means “broken”; it means “broken, or the test is just being flaky again.” Once a human has to interpret a red build, the test has stopped being an automated check and become a coin flip that wastes your attention. And the damage does not stay local. The moment one test is known to cry wolf, people start reflexively re-running the whole suite on any red — including real failures. The flakiness of one test erodes trust in every test around it.

   Healthy suite:   red ─► "something is broken"      ─► investigate
   Flaky suite:     red ─► "ugh, is it real this time?" ─► re-run, ignore
                                                          └► real bugs slip through

So the cost of flakiness is not the seconds spent re-running. It is the slow conversion of a confidence-machine into background noise. No test at all is honest: you know you have no coverage there. A flaky test lies: it claims coverage, then withdraws it at random.

Suppose each test in your suite is independently “flaky-green” 99.5% of the time (fails spuriously 1 run in 200 — which feels rare per test). Then a suite’s chance of a clean run is 0.995^N:

  N tests     P(whole suite passes)   ≈ spurious-red rate
     50         0.995^50  = 0.778          22% of runs
    200         0.995^200 = 0.367          63% of runs
    800         0.995^800 = 0.018          98% of runs

At 800 tests, a suite where each individual test is “99.5% reliable” almost never has a clean run. Flakiness that is invisible per-test becomes a permanent red build in aggregate. This is why teams treat a single flaky test as a fire, not a nuisance.

The four families of flakiness

Nearly every flaky test traces to one of four root causes. Each has a distinct signature — a pattern in how and when it fails — and learning to read the signature is most of the diagnosis.

   ┌──────────────┬───────────────────────────┬────────────────────────────┐
   │ Family       │ Root cause                │ Tell-tale signature        │
   ├──────────────┼───────────────────────────┼────────────────────────────┤
   │ Timing/async │ racing an event vs a clock│ fails on slow/loaded CI,   │
   │              │                           │ passes locally             │
   │ Order        │ tests leak state → next   │ fails only in some run     │
   │ dependence   │ test                      │ order; passes when isolated│
   │ Shared state │ DB/singleton/file/cache   │ fails under parallelism;   │
   │              │ not reset                 │ passes run alone           │
   │ External     │ real net / 3rd-party /    │ fails at random, in bursts,│
   │              │ clock / locale            │ correlated with the world  │
   └──────────────┴───────────────────────────┴────────────────────────────┘

Family 1 — Timing and async races

This is the biggest single source of flakiness in integration and E2E suites, because these tests deal in events that happen eventually rather than values that exist now. You kick off an action — send a request, enqueue a job, click a button — and the result arrives on its own schedule. The test has to decide when to check for it, and that decision is where the flake lives.

The fixed sleep. The classic broken pattern is to wait a hard-coded duration and hope it was long enough:

start_background_job()
time.sleep(2)          # "2 seconds is usually enough"
assert job_is_done()   # ...until the CI box is loaded and it takes 2.3s

The word usually is the tell. sleep(2) encodes an assumption — “the job finishes in under two seconds” — that is true on your laptop and false on a shared CI runner under load. When it is too short, the test fails though the code is fine (a false red). When you “fix” it by bumping to sleep(10), every run now pays ten seconds whether it needs them or not, and the suite rots into slowness. A fixed sleep is always either too short (flaky) or too long (slow), and often both on different machines.

The race. Even without an explicit sleep, any two concurrent operations whose relative order is not enforced can interleave differently between runs. A test that assumes writer-then-reader ordering will pass whenever the scheduler happens to run them that way and fail when it does not. The real Rust test in rust/kvlite/tests/concurrency.rs is instructive precisely because it is not flaky: eight threads hammer one shared store, but the test asserts only on a quantity that is order-independent — the final count of distinct keys — so no interleaving can change the answer.

// Not flaky: the assertion is invariant under thread interleaving.
// THREADS * PER_THREAD distinct keys go in; exactly that many must exist.
assert_eq!(store.len().unwrap(), THREADS * PER_THREAD);

The lesson generalizes: a concurrent test is only stable if what it asserts is invariant under every legal ordering. Assert on an order-dependent detail and you have written a race into your test.

Signature: passes locally, fails on CI; fails more when the machine is loaded; “fixes itself” when you add or lengthen a sleep; fails more often the more parallel work is running.

Under the hood — why CI amplifies timing flakiness

A timing flake that never fires on your laptop and fires constantly on CI is not a coincidence, and understanding the mechanism tells you where to look. Three properties of CI environments systematically stretch the gap between “I started the action” and “the result is ready,” which is exactly the window a fixed sleep is gambling on:

Shared, oversubscribed CPUs. CI runners are often small virtual machines packed onto busy hosts. Your test thread does not get a dedicated core; it competes with other jobs, so wall-clock time for the same work balloons unpredictably. The 2-second job takes 2.4 seconds because it spent 0.4 seconds waiting for a CPU it did not get.
Cold caches and first-touch costs. Locally, the JIT is warm, the disk cache is hot, the connection pool is full. A fresh CI container pays every one of those costs on the first run, so the first time a test does something is its slowest — and CI does everything for the first time.
The noisy-neighbour spike. Even a fast runner has moments where an unrelated process steals the disk or the network for a few hundred milliseconds. Those spikes are rare, which is why the test is flaky and not simply broken: it fails on the small fraction of runs that happen to coincide with a spike.

   Laptop timeline:  |start|===work 0.3s===|done|  sleep(2) has huge slack → green
   CI timeline:      |start|== wait CPU ==|=work=|== disk spike ==|done|
                                                        sleep(2) undershoots → red

The takeaway for diagnosis: if a test only flakes on CI and the flake tracks machine load, you are almost certainly looking at Family 1. The environment did not introduce a new bug; it widened a timing window your test was already gambling on.

Family 2 — Order dependence

An order-dependent test passes only when the suite runs its tests in a particular sequence, because an earlier test leaked state that a later test silently depends on — or that a later test is broken by.

   test_a: creates user "alice" in the DB, never deletes her
   test_b: asserts "the users table has exactly 1 row"

   run [a, b] → b sees alice + its own row → 2 rows → FAIL
   run [b, a] → b runs first, sees only its own row → PASS
   run [b]    → PASS

Nothing is wrong with test_b in isolation. It is wrong because test_a mutated a world they share and did not clean up. The insidious part: this can stay hidden for months, until someone renames a test, adds a new one, or a runner randomizes order — and the build goes red with a diff that touches neither failing test. Many modern runners (e.g. randomized ordering in Go’s test shuffling, or pytest-randomly) deliberately randomize order to surface these dependencies early rather than letting them hide.

Signature: the test passes when run alone (pytest path::test_b), fails as part of the full suite; the failure appears or disappears when test order changes; the failing test’s own code was not touched by the commit that turned it red.

There is a subtle second flavour worth naming: hidden ordering guarantees you are relying on by accident. Sometimes test_b does not depend on test_a’s data but on its side effect — a migration test_a happened to run, a directory it created, a global it initialized. test_b “passes in the suite” only because some earlier test did that setup for it. Run test_b alone and it fails for the opposite reason: the world it silently assumed was never built. Both flavours are the same bug — a test that is not self-contained — and both are cured the same way, by making every test set up and tear down its own world.

Family 3 — Shared state

Order dependence is one symptom of a deeper disease: shared mutable state that is not reset between tests. Any resource that outlives a single test and can be mutated is a shared channel through which one test can contaminate another:

Databases — rows one test inserts are visible to the next until something truncates them.
Singletons and global variables — an in-memory cache, a configured logger, a connection pool, a static mut counter. The first test to touch it sets state the rest inherit.
Files and directories — a test that writes /tmp/output.json and the next that reads it, or two tests writing the same temp path.
Caches — a memoized value computed in one test served stale to another.
External services and ports — a fixed port a test binds, a shared queue, a shared bucket. Two tests that both grab port 8080 collide the moment they run at once.

Shared state is usually dormant — it hides as harmless order dependence — until you introduce parallelism, at which point it detonates. Run tests one at a time and the writes at least happen in some deterministic order. Run them concurrently across workers and two tests now write the same row, the same key, the same file at the same time, and the winner is decided by the scheduler:

   Serial (dormant bug):        Parallel (bug fires):

   worker: A writes k=1              worker 1: A writes k=1  ┐ race on
           A reads  k → 1            worker 2: B writes k=2  ┘ the same key
           B writes k=2
           B reads  k → 2            A then reads k → 2 (!!)  ─► FAIL

This is why “it only fails when I run with -j 8” is the classic shared-state signature. Parallelism does not create the bug; it exposes a shared resource the tests were always fighting over, by removing the accidental serialization that hid the fight.

Signature: passes serially, fails under parallel execution; the failure rate rises with the worker count; two unrelated tests fail together; re-running just the failures makes them pass (because now they are serial again).

Family 4 — External flakiness (the world moves)

The final family comes from outside your process entirely. When a test reaches across the network or reads the real environment, it inherits the nondeterminism of the whole world.

Real network calls — packets drop, latency spikes, a load balancer resets a connection. Every real hop is a small independent chance of failure that the test did not intend to assert on.
Third-party APIs — a service you don’t control returns 503, rate-limits your 20 rapid test requests, changes a field, or is simply down for maintenance during your CI run.
DNS — a name lookup that usually takes 5 ms occasionally takes 5 seconds and blows your timeout, or a resolver returns a stale record.
Clock, timezone, and locale drift — the sneakiest of all, because they fail deterministically at the wrong moment. A test that computes “tomorrow” and formats it can pass for 364 days and fail on the day a DST transition makes “tomorrow” 23 hours long. A test that sorts names passes in the CI’s C locale and fails on a developer’s machine set to a locale that orders accented characters differently. A test that parses 03/04 as a date is right in one country and wrong in another.

There is a compounding effect that makes external flakiness feel worse than any single dependency’s reliability suggests. If a test crosses k independent real hops — DNS, then a load balancer, then a third-party API, then their database — and each is up 99.9% of the time, the test’s own reliability is the product:

   1 hop:  0.999^1 = 0.9990   →  1 spurious fail in ~1,000 runs
   4 hops: 0.999^4 = 0.9960   →  1 in ~250
   8 hops: 0.999^8 = 0.9920   →  1 in ~125

Every real dependency you add to a test multiplies in another small chance of a failure you never intended to assert on. This is the deep reason the pyramid keeps high-fidelity, many-hop tests few: their unreliability compounds with each real hop.

Signature: fails at random and often in bursts (the third-party API had a bad ten minutes, so five unrelated tests failed together); failures correlate with the outside world — the time of day, a specific date boundary, a particular CI region, or a locale — rather than with your code or your test order.

A whole class of clock-driven bugs shares one root: a program assumes time fits in a specific representation. The best-known is the Year 2038 problem — systems that store Unix time as a signed 32-bit integer overflow just after 03:14:07 UTC on 19 January 2038, wrapping to a negative number and computing dates in 1901. It is deterministic, not random: the code is always wrong past that instant. The testing lesson is the general one behind Family 4. Time, timezone, and locale are inputs your test rarely varies, so a test that hard-codes “now” or trusts the machine’s clock can pass for years and then fail catastrophically at one specific boundary — a leap day, a DST switch, a 32-bit rollover — that no one thought to sample. Treat the clock as a dependency to be injected and varied, not an ambient fact to be trusted.

The one diagnostic question

When a test goes red and you suspect flakiness, you do not need to guess among four families. You need one experiment: can I make it fail on demand? Run it in isolation, then in the full suite, then serially, then in parallel, then at a different simulated time. Whichever axis flips the result is the family:

   passes alone, fails in suite ............... order dependence / shared state
   passes serial, fails parallel .............. shared state
   passes fast, fails on loaded/slow CI ....... timing / async race
   fails in bursts, tied to time/net/locale ... external

One warning about this experiment: flakes are, by definition, probabilistic, so a single passing run proves nothing. To trust a “passes alone” result you must run it many times — dozens, sometimes hundreds — before concluding an axis does not flip it. A flake that fires 1 run in 50 will look perfectly stable across your first 10 attempts. Treat “I couldn’t reproduce it” as “I haven’t run it enough,” not “it’s fine.”

The cure follows from the family, and that is the whole of the next page. But you cannot cure what you cannot name — and now you can name all four.

The architect’s lens

Why does the concept of flakiness exist? Because as tests climb from pure functions to real systems, they inherit nondeterminism — concurrency, shared resources, real time, and the network — and a test built on a nondeterministic foundation can disagree with itself run to run. Flakiness is the name for that self-disagreement.
What problem does naming it solve? It converts an infuriating “the build is randomly red” into a diagnosable engineering problem with four known causes and four known signatures, so you can fix the test instead of endlessly re-running it.
What are the trade-offs? The conditions that cause flakiness — real dependencies, parallelism, real time — are the same conditions that give integration and E2E tests their fidelity. You cannot eliminate flakiness by lowering fidelity to zero without giving up the boundary coverage that motivated the test.
When should I avoid treating a red as flaky? Always, until proven. The dangerous move is to assume flakiness and re-run; that habit is exactly how a real regression gets waved through. Reproduce and classify first; only a confirmed, understood flake earns a re-run.
What breaks if I remove the discipline of hunting flakiness? Trust. A suite with tolerated flaky tests trains everyone to ignore red, at which point the suite stops buying confidence and becomes a slow, noisy tax that real bugs sail straight through.

Check your understanding

Give the one-sentence definition of a flaky test, and explain the argument for why a flaky test is worse than having no test at all.
A test uses time.sleep(2) before asserting a background job is done. Name the two distinct ways this line fails you, and why bumping it to sleep(30) is not a real fix.
test_b passes when you run it alone but fails as part of the full suite, and the commit that turned it red did not touch test_b at all. Which family is this, what is the underlying mechanism, and what one experiment confirms it?
Why does shared state so often stay hidden until you turn on parallel test execution? Use the terms dormant and serialization in your answer.
A date-formatting test passes for months, then fails on one specific calendar day and never before. Which family is this, and what does the Year 2038 example teach about the general shape of the bug?

Show answers

A flaky test passes and fails on the same code, with no change to the test or the system under test — only to invisible surrounding conditions. It is worse than no test because it destroys the meaning of red: red no longer means “broken,” it means “broken or just flaky,” which forces humans to interpret every red build, trains them to re-run and ignore failures, and thereby erodes trust in the whole suite. No test is at least honest about having no coverage; a flaky test lies about having it.
(a) Too short: on a slow or loaded machine the job isn’t done yet, so the test fails though the code is correct — a false red. (b) Too long / slow: every run pays the full duration even when the job finished instantly, rotting the suite into slowness. sleep(30) only trades the first failure for the second: it is still a hard-coded guess, now guaranteed to waste 30 seconds every run, and still fails if a job ever legitimately takes longer. The fix is to wait on the condition, not the clock.
Order dependence (a symptom of shared state). An earlier test leaked mutable state — a DB row, a global, a file — that test_b implicitly depends on or is broken by; the red-turning commit merely changed the run order or added a test. Confirm it by running test_b in isolation (it passes) versus in the suite (it fails), and by shuffling test order to make the failure appear or disappear.
Run serially, tests touch a shared resource in some deterministic order, so the contamination is dormant — it looks at worst like benign order dependence. Parallel execution removes the accidental serialization that hid the conflict: two tests now write the same row/key/file simultaneously, and the scheduler decides the winner, so the latent shared-state bug finally fires. Parallelism exposes the bug rather than creating it.
External flakiness — specifically clock/timezone/locale drift. The Year 2038 example shows the general shape: time and locale are inputs the test rarely varies, so a test that trusts the ambient clock can be correct for a long time and then fail deterministically at one un-sampled boundary (a leap day, a DST transition, a 32-bit rollover). The fix in spirit is to treat the clock as an injectable dependency and test the boundaries explicitly, rather than trusting “now.”