Skip to content
Software Testing · First Principles

Determinism: Controlling Time, Randomness & Order

The previous page ended on a warning: the moment one test depends on data another test left behind, you have “manufactured a flaky test out of thin air.” This page is about that word — flaky — because it is the single most corrosive failure mode a test suite can have. Bad data ownership is only one of its causes. Time, randomness, async races, and ordering assumptions are the rest.

A failing test that is right is a gift: it caught a bug. A failing test that is flaky is a slow poison: it teaches the team to stop believing the color of the build. This page is about earning back that belief by making tests do the same thing every single time they run.

What flakiness is, and why one flaky test is contagious

Section titled “What flakiness is, and why one flaky test is contagious”

A flaky test is a test that passes or fails without any change to the code under test. Run it twice on the same commit; get two different answers. It is nondeterministic — its result depends on something other than the code it claims to verify.

That is not merely annoying. It attacks the entire premise of testing. The whole reason to run a suite is this deduction:

   green  →  "the change is safe"
   red    →  "the change is broken, stop"

That deduction only holds if red means broken. A flaky test breaks the implication. Now red might mean “broken” or might mean “the CI box was busy for 200 ms and a timer fired late.” A human cannot tell the two apart from the outside — so they do the rational thing: they re-run the job. It goes green. They merge.

And there is the poison. Once re-running until green is normal, red has stopped meaning anything. The team has learned to ignore failures. The next time a real bug turns the build red, it looks exactly like the flake they’ve been re-running past all week — so it ships. One flaky test doesn’t cost you one test; it quietly disarms every other test in the suite, because it trains everyone to treat red as noise. A suite you ignore is worse than no suite: it has all the cost and none of the confidence.

   one flaky test
        │
        ▼
   "just re-run it"  ──►  red no longer means "stop"
        │
        ▼
   real bugs look like flakes  ──►  they ship  ──►  suite is decorative

So flakiness is not a minor-severity bug in a test. It is a trust bug in the whole suite, and it deserves to be treated as an incident, not a nuisance.

Almost all flakiness traces to one root cause: a test consumed a value from the outside world that it did not control. The cure, in every case, is the same shape — stop reading the uncontrolled thing; inject a controlled one. The three big offenders are time, randomness, and order.

Control time: never call the real clock

Section titled “Control time: never call the real clock”

The wall clock is the most common hidden input in a test suite, because time-dependent logic is everywhere: token expiry, cache TTLs, “created less than 24 hours ago,” retry backoff, scheduled jobs. If that logic reads the system clock directly, the test is at the mercy of when it happens to run.

# Flaky by construction: reads the real clock, twice, in two places.
def is_expired(token):
    return token.created_at + TTL < datetime.now()   # ← hidden input


def test_token_expires():
    token = make_token()          # created_at = now()  (call #1)
    time.sleep(TTL + 1)           # slow AND fragile
    assert is_expired(token)      # compares against now() (call #2)

That test is slow (it really sleeps) and nondeterministic (it assumes the two now() calls are far enough apart, which a loaded CI box can violate). The problem is not the test — it is that production code reached out and grabbed now() itself.

The fix is a first-principles one: make time an input, not an ambient fact. Inject a clock. The code under test asks a clock it was given for the time; production wires in the real system clock, tests wire in a fake one they can set to any instant.

# Time is now a dependency, so a test can freeze it.
def is_expired(token, clock):
    return token.created_at + TTL < clock.now()


def test_token_expires():
    clock = FakeClock(at="2026-07-08T12:00:00Z")
    token = make_token(created_at=clock.now())
    clock.advance(TTL + 1)                 # instant; no real sleeping
    assert is_expired(token, clock)        # deterministic, and fast

Three things improved at once. The test is fast (advancing a fake clock is free), deterministic (the same instants every run), and more powerful (you can jump to any time — a leap second, the year 2038, the daylight-saving boundary — which a real sleep could never reach). Most test frameworks ship “fake timers” or a “freeze time” helper for exactly this; a hand-rolled injectable clock does the same job and works in any language.

Under the hood — three ways to fake a clock

Section titled “Under the hood — three ways to fake a clock”

There is a spectrum, from most invasive to cleanest:

  monkey-patch now()        fake timer library        injected clock
  ─────────────────         ──────────────────         ──────────────
  overwrite the global      framework replaces the     code asks a Clock
  time function at runtime   scheduler's time source    object it was given
  works, but spooky         convenient, framework-     explicit, testable,
  action at a distance      specific                    portable, no magic

Monkey-patching (globally replacing datetime.now) works but is a hidden global mutation — the very thing the order section warns against, since a patch that leaks past one test poisons the next. Fake-timer libraries are convenient and control not just “now” but scheduled callbacks and timeouts. The injected clock is the most honest: time appears in the function signature, so you cannot forget it’s a dependency. Prefer injection where you can; reach for framework timers where the code is already written against the clock directly.

Control randomness: seed it or inject it

Section titled “Control randomness: seed it or inject it”

The second uncontrolled input is randomness. A test that exercises code using a random UUID, a shuffled list, a random sample, or a jittered retry delay will, by definition, do something different every run — and when it fails, it fails on a value you can never see again.

The principle is identical to time: do not read the global random source; control it. Two ways:

  • Seed the RNG. Pseudo-random generators are deterministic functions of their seed. Fix the seed and the “random” sequence is exactly reproducible — the shuffle, the sample, the UUIDs come out the same every run. Failures reproduce on demand.
  • Inject the value. Where the randomness is a single decision (“pick a winner,” “generate an id”), pass in a chooser the test can pin, exactly like the clock.
# Seeded: the "random" sequence is fixed and reproducible.
def test_shuffle_is_stable_under_seed():
    rng = Random(seed=1234)          # fixed seed → fixed sequence
    deck = shuffle(range(52), rng)
    assert deck[0] == 7              # true for THIS seed, every run


# Injected: the decision is handed in, so the test pins it.
def test_lottery_picks_the_injected_winner():
    winner = pick_winner(entrants, chooser=lambda xs: xs[0])
    assert winner == entrants[0]     # no randomness left to flake

There is a subtlety worth stating plainly. Seeding makes a failure reproducible, which is what you need for debugging — but a test that only ever runs one fixed seed also only ever tests one path through the randomness. That is fine for most tests. The deliberate exception is property-based / fuzz testing, which intentionally runs many random inputs to find edge cases — and the good frameworks square this circle by printing the failing seed so any failure is instantly replayable. Randomness for coverage is a feature; randomness you can’t reproduce is the bug.

Eliminate order dependence: any order, any parallelism

Section titled “Eliminate order dependence: any order, any parallelism”

The third source is the sneakiest, because each test can look perfectly deterministic on its own. Order dependence is when a test’s result depends on which other tests ran before it, or alongside it. The suite passes in the order you happen to run it, and fails the day someone shards it across four workers or the runner shuffles test order.

The governing rule is absolute: every test must pass in any order, and in parallel. A suite that only passes in one specific sequence is not a suite; it is one big fragile test wearing many hats.

Order dependence has a small, well-known cast of culprits:

  • Shared mutable state. Two tests touch the same row, file, cache key, or singleton. Test A leaves it dirty; test B assumed it clean. (This is the mystery-guest / ownership problem from the previous page, seen from the ordering side.)
  • Leaked globals. A test sets an env var, a feature flag, a monkey-patch, a global config — and forgets to restore it. Every later test runs in the contaminated world.
  • Ordering assumptions. A test relies on the nth test having created a record it needs, or on ids being sequential, or on “the first test seeds the DB.”

The kvlite integration tests in this repo show the discipline that makes order irrelevant: each test manufactures a store nobody else can see.

// rust/kvlite/tests/concurrency.rs — a per-test path no other test can touch.
static N: AtomicU64 = AtomicU64::new(0);
let path = std::env::temp_dir().join(format!(
    "kvlite-conc-{}-{}.wal",
    std::process::id(),                       // unique to this process/run
    N.fetch_add(1, Ordering::Relaxed)         // unique to this test
));
let _ = std::fs::remove_file(&path);          // start from nothing
let (db, _) = Db::open(&path).unwrap();

Because the path embeds the process id and an atomic counter, no two tests — even the 8 threads that same file hammers in the concurrency test — can ever collide. There is no shared state to leave dirty, so there is nothing for order to depend on. Cargo runs Rust tests in parallel by default, which is not a hazard here but a free flakiness detector: parallel execution surfaces shared-state bugs that a serial run would hide. Running your suite in a randomized order (many frameworks offer a --shuffle/--random-order flag) does the same job on purpose — if shuffling turns the suite red, you have an order dependency to hunt down, not a flag to disable.

The rest of the flakiness zoo

Section titled “The rest of the flakiness zoo”

Time, randomness, and order are the big three, but a few more hide in the corners. Learn to recognize them:

  • Async timing and races. awaiting the wrong thing, a fixed sleep(100) that “usually” is long enough, two callbacks whose order isn’t guaranteed. The fix is to wait on a condition (poll until the state is true) rather than a duration, and to make concurrent operations deterministic where you can.
  • Real network calls. A test that hits a live API, DNS, or a third-party service inherits their uptime, latency, and rate limits as its own flakiness. Stub or mock the boundary; test against the real thing only in a small, clearly-labelled integration tier.
  • Unclosed resources. A file handle, socket, DB connection, or thread left open by one test can make a later, unrelated test fail when the OS runs out of descriptors or a port is still bound. Clean up in a guaranteed teardown.
  • Timezone and locale assumptions. Code (or a test) that assumes UTC, a . decimal separator, or ASCII sorting passes on the author’s machine and fails in CI set to another region. Pin the timezone and locale explicitly; don’t inherit the host’s.

Every one of these is the same disease as the big three: an uncontrolled input from the environment. The cure is always to name it and pin it.

A strategy for a flaky test: quarantine, reproduce, fix the root

Section titled “A strategy for a flaky test: quarantine, reproduce, fix the root”

When you find a flaky test, resist the two tempting non-fixes. Do not just add a retry (“re-run failed tests up to 3 times”) — that hides the flake instead of removing it, and a retried test can no longer catch a real intermittent bug, because it’ll paper over that too. Do not just delete the test — you’d be discarding whatever coverage it had, to buy quiet. Both trade real confidence for a green light. Work the problem instead:

  1. QUARANTINE     move it out of the blocking suite so it can't
                    train the team to ignore red — but TRACK it,
                    don't forget it. Quarantine is a hospital, not a grave.


  2. REPRODUCE      make it fail on demand. Run it 1000x; run it in a
                    shuffled/parallel suite; pin then vary the seed, the
                    clock, the timezone. A flake you can't reproduce,
                    you can't prove you've fixed.


  3. FIX THE ROOT   name the uncontrolled input — clock, RNG, shared
                    state, race, network — and inject/pin/isolate it.
                    Then un-quarantine and prove it's stable.

Quarantine buys the suite’s trust back immediately (the blocking build is deterministic again) without pretending the bug is gone (it’s tracked, with an owner and a deadline). Reproduction is the whole game: nondeterminism feels like magic only until you control the one input that was varying, at which point it becomes boring and fixable. And fixing the root cause — not the symptom — is what keeps the flake from resurrecting under a new name next month.

The architect’s lens

Section titled “The architect’s lens”
  • Why does it exist? Because tests silently inherit uncontrolled inputs — the clock, the RNG, execution order, the network — and any uncontrolled input makes a test’s result depend on something other than the code under test. Determinism is the discipline of removing those inputs.
  • What problem does it solve? It preserves the one deduction the whole suite rests on — red means broken — by making every test do the same thing every run, so failures are trustworthy and reproducible.
  • What are the trade-offs? Determinism costs design effort: you must inject clocks, seed RNGs, isolate state, and mock boundaries instead of writing the quick version that reads globals. You also give up the accidental coverage that “random” runs sometimes provide (recovered deliberately via seeded property tests).
  • When should I avoid it? Almost never for unit and integration tests. The deliberate exception is property-based/fuzz and load testing, where varied random input is the point — but even there you keep it reproducible by recording and printing the seed.
  • What breaks if I remove it? Flakiness compounds across the suite; teams learn to re-run past red; real bugs start to look like flakes and ship. The suite keeps all its cost and loses all its confidence — the worst possible state for a test suite to be in.

Check your understanding

Section titled “Check your understanding”
  1. Define a flaky test precisely, and explain the chain by which one flaky test can undermine confidence in an entire suite.
  2. A test calls time.sleep(TTL + 1) and then asserts a token is expired. Name both problems with it and describe the single design change that fixes both.
  3. Your team relies on random inputs to catch edge cases, but a colleague says “randomness makes tests flaky, ban it.” Reconcile these: how do you get the coverage benefit of randomness while keeping failures reproducible?
  4. Give three distinct culprits behind an order-dependent test, and explain why running the suite in parallel or in a shuffled order is a useful tool rather than a hazard.
  5. A test flakes about 1 in 200 runs. Walk through the correct three-step response, and say specifically why “wrap it in a retry” and “delete it” are both wrong.
Show answers
  1. A flaky test passes or fails without any change to the code under test — its result depends on something other than that code (nondeterminism). The chain: red should mean “broken, stop,” but a flake makes red ambiguous → people re-run until green and merge → re-running-past-red becomes the norm → a real bug’s red now looks identical to the flakes everyone ignores → it ships. One flaky test trains the whole team to treat red as noise, disarming every other test.
  2. It is slow (it really sleeps for the whole TTL) and nondeterministic (it assumes the two now() reads are far enough apart, which a loaded box can violate). The single fix is to inject a clock: the code asks a Clock it was given for the time, and the test freezes/advances a fake clock instantly to any instant — fast and deterministic at once.
  3. They’re both right about different scopes. Use randomness deliberately for coverage (property-based / fuzz tests that run many inputs to find edge cases), but keep every run reproducible by fixing or recording the seed — good frameworks print the failing seed so any failure replays exactly. Reproducible randomness is a feature; randomness you can’t reproduce is the bug the colleague means.
  4. Any three of: shared mutable state (a row/file/cache/singleton left dirty by one test and assumed clean by another), leaked globals (an env var, feature flag, or monkey-patch set and not restored), ordering assumptions (relying on an earlier test to have created a record, or on sequential ids). Parallel/shuffled execution is a flakiness detector: it deliberately breaks any hidden ordering assumption, so a suite that only passes in one sequence goes red in your CI instead of silently in production — surfacing the bug rather than hiding it.
  5. Quarantine (move it out of the blocking suite so it stops training people to ignore red — but track it with an owner, don’t forget it) → Reproduce (make it fail on demand: run it many times, shuffled/parallel, pinning then varying clock/seed/timezone) → Fix the root cause (name the uncontrolled input and inject/pin/isolate it, then un-quarantine and prove stability). A retry hides the flake and destroys the test’s ability to catch real intermittent bugs; deleting it throws away real coverage. Both buy a green light by spending confidence — the opposite of the goal.