The FIRST Properties of a Good Test

In What Is a Unit? Arrange-Act-Assert you learned to shape a test: arrange a known world, act once, assert on the result. But shape alone does not make a test worth keeping. You can write a perfectly-shaped test that is so slow no one runs it, so entangled with its neighbours that reordering the file breaks it, or so vague that it “passes” whether the code works or not. A test like that is worse than no test: it costs time, it lies, and eventually the team learns to ignore it.

This page is about the difference between a test you trust and a test you learn to route around. There is a durable checklist for that difference, and it spells a word: FIRST — Fast, Isolated, Repeatable, Self-validating, and Timely. These five properties are not style preferences. Each one, when violated, produces a specific, recognizable failure mode. Learn the five, and you can diagnose a bad test on sight.

  F  Fast            runs in milliseconds → runs on every save
  I  Isolated        no order, no shared state → any subset passes alone
  R  Repeatable      same input → same result, every machine, every run
  S  Self-validating a green/red assertion → no human reads the output
  T  Timely          written next to the code → behaviour pinned while cheap

F — Fast

A unit test should run in milliseconds. Not “fast enough” — fast enough that the entire suite finishes in the time it takes you to glance away from the keyboard. This is not vanity. Speed is what lets you run the whole suite on every save. A suite that runs in 200 ms is a background reflex; a suite that takes two minutes is a decision, and a decision you will increasingly decide not to make.

The reason unit tests can be this fast is that a unit — as we defined it last page — touches no network, no disk-with-fsync, no clock, no other process. It exercises a function in memory. The moment a test reaches out of the process, it inherits that thing’s latency: a TCP connect is tenths of a millisecond at best and tens of milliseconds under load; a real fsync to disk is milliseconds; a real HTTP call is a coin flip.

You can see the contrast inside this repo. The kvlite integration tests in rust/kvlite/tests/server.rs bind a real TcpListener, spawn the server on a background thread, and drive it with a real client socket:

let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
thread::spawn(move || {
    let _ = server::serve_on(listener, db);
});
let stream = TcpStream::connect(addr).unwrap();

That test is slower by design — and that is correct, because its job is to prove the wire protocol works end to end, which a fast in-memory test cannot. The lesson is not “TCP tests are bad.” It is that they belong to a different tier with a different budget. Your unit tests — the ones exercising Db::set / Db::get in memory — should have no TcpStream, no thread::spawn, and should finish before you notice.

Assume a suite of 2,000 tests.

  In-memory unit test   ≈ 0.1 ms each  → 2,000 × 0.1 ms   = 0.2 s   (runs on save)
  TCP round-trip test   ≈ 5   ms each  → 2,000 × 5   ms   = 10  s   (runs on push)
  Real HTTP + DB test   ≈ 80  ms each  → 2,000 × 80  ms   = 160 s   (runs in CI, maybe)

A 0.2 s suite runs ~50 times an hour, unnoticed.
A 160 s suite runs ~twice a day, resented.
The feedback loop is the product. Speed is not a nice-to-have; it decides how often the test runs at all.

Failure mode: a slow test is one that is not run. It quietly stops protecting you long before it turns red.

I — Isolated

Each test must stand on its own. It must not depend on running after some other test, and it must not depend on state another test left behind. The blunt check: pick any single test, run it alone, and it passes. Run the file in reverse order, and everything still passes. If either fails, your tests are coupled.

The usual culprit is shared mutable state — a file, a global, a database row — that two tests both touch. Test A leaves a key in the store; Test B assumes the store is empty; B passes only because A ran first. Now someone runs B alone, or the runner shuffles order, and B fails for a reason that has nothing to do with the code under test. You have manufactured a lie.

The repo shows the fix directly. Both kvlite server tests persist to a write-ahead log file, which is shared, mutable, on-disk state — exactly the isolation hazard. So each test does not reuse one path; it mints a unique one:

fn temp_wal() -> std::path::PathBuf {
    static N: AtomicU64 = AtomicU64::new(0);
    std::env::temp_dir().join(format!(
        "kvlite-srv-{}-{}.wal",
        std::process::id(),          // unique per test run (this process)
        N.fetch_add(1, Ordering::Relaxed) // unique per call within the run
    ))
}

Read what makes that path unique. The process id separates one cargo test run from any other running concurrently or a stale file from yesterday. The atomic counter hands every call a fresh integer, so two tests in the same run can never collide. The result: each test gets its own file, opens its own Db, and cannot see or corrupt a sibling’s state. Order does not matter; parallelism does not matter. That is isolation made physical — not a comment saying “please don’t share,” but a construction that cannot share.

  Shared state (coupled)            Isolated
  ┌──────────────┐                  ┌────────┐  ┌────────┐  ┌────────┐
  │  one.wal     │                  │ w-0.wal│  │ w-1.wal│  │ w-2.wal│
  └───┬─────┬────┘                  └───┬────┘  └───┬────┘  └───┬────┘
    testA  testB  ← order matters      testA      testB      testC
                                       (any order, any subset, all green)

Failure mode: an order-dependent test. It passes in the file but fails alone, or fails when the runner shuffles — a bug in the test, masquerading as a bug in the code.

R — Repeatable

Run the test today, tomorrow, on your laptop, on a colleague’s machine, in CI — same input, same result, every time. A test that is green on your machine and red in CI teaches the team that red doesn’t mean broken, and once red is ambiguous, the whole suite loses authority.

Non-repeatability creeps in through three doors, all of them ambient machine state the test forgot it depends on:

Wall-clock time. A test that computes “is this token expired?” against now() will pass until it runs at the wrong second, or in a different time zone, or on a leap day. Inject the clock; pass a fixed instant.
The network. A test that hits a real endpoint depends on that endpoint, on DNS, and on the coffee-shop Wi-Fi. It is testing the internet, not your code. (We isolate these dependencies with seams in Isolating Dependencies With Seams.)
Randomness and ordering. An unseeded random value, an un-sorted hash-map iteration, a race between threads — any of these makes the outcome a dice roll.

The word for a test whose result changes without the code changing is flaky, and flaky is not bad luck. It is a repeatability violation with a specific hidden input — a clock, a socket, a seed, a race. Find the input, pin it, and the flake disappears. A flaky test is a diagnosis, not a weather report.

S — Self-validating

A test must decide its own verdict: pass or fail, green or red, with no human reading the output. That means a real assertion — assert_eq!, matches!, assert! — that the runner can check. It does not mean printing a value and eyeballing it.

Compare the two. This is not a test, no matter how it looks:

let reply = round_trip(&mut r, &mut w, "GET name");
println!("{reply}"); // "hmm, looks right?"  ← who checks this? when?

This is a test, because it has an opinion the machine can enforce:

assert_eq!(round_trip(&mut r, &mut w, "GET name"), "VALUE ada lovelace");
assert!(round_trip(&mut r, &mut w, "bogus").starts_with("ERR"));

Both lines are lifted from rust/kvlite/tests/server.rs. Each one nails down an exact expected value; the runner compares, and a mismatch is a failure with a diff, not a paragraph a tired human is supposed to scan at 5 p.m. matches! is the pattern-matching cousin — use it when you care about the shape of a result (assert!(matches!(parse(x), Err(ParseError::Empty)))) rather than an exact equal.

The tell for a self-validation failure is a test that only “works” when someone is watching — logs to read, a screenshot to compare, a “yeah that seems fine.” If a human is in the assertion loop, the test does not scale past that human’s attention, and CI cannot run it at all.

Failure mode: a test that prints instead of asserts. It is green forever, including when the code is broken, because nothing ever checks it.

T — Timely

Write the test close to the code — ideally alongside it, ideally before it. The value of a test is highest while the behaviour is fresh and the design is still soft. That is when a test is cheapest to write (you remember every branch) and most useful (it can still change the shape of the code it describes).

The cost curve runs the other way. Six months later, that same behaviour is a black box: you no longer remember which edge cases mattered, the code has grown dependencies that make it hard to test, and writing the test now means first untangling the code to make it testable. The behaviour was cheap to pin the day it was written and expensive to pin ever after. “Timely” is the property that says: pin it while it’s cheap.

Writing the test first (the discipline we return to in Tests as Living Documentation) has a second effect — it forces you to design code you can test. Code written test-last is often code that was never asked whether it was testable, which is why so much legacy code needs seams retrofitted before a single assertion can be written.

Roughly 37 seconds after launch, the maiden Ariane 5 flight self-destructed. The proximate cause was a piece of inertial-reference software reused from Ariane 4: a 64-bit floating-point horizontal-velocity value was converted to a 16-bit integer, and Ariane 5’s higher velocity overflowed the 16 bits. The conversion had no guard because on Ariane 4 that value could never get large enough to overflow — an assumption that was true, then silently wasn’t.

The testing lesson is FIRST in miniature. The reused component was never re-tested against Ariane 5’s actual flight envelope (Timely — the test that mattered was never written for the new context), and the overflow path had no assertion that would have gone red on the new inputs (Self-validating). A cheap, timely test that fed in the real velocity range and asserted on the conversion would have caught on the ground what cost a rocket in the air. (Details simplified; the lesson is exact.)

Diagnosing smells with FIRST

The point of the checklist is that it turns vague complaints into precise diagnoses. “This test is annoying” is not actionable. FIRST makes it so:

  Symptom                                   FIRST property violated   The real fix
  ─────────────────────────────────────────────────────────────────────────────────────
  "The suite takes too long, we skip it"    Fast                      remove the I/O; move to a slower tier
  "Passes in the file, fails run alone"     Isolated                  give it its own state (unique temp path)
  "Green here, red in CI, then green again"  Repeatable               find the clock/network/seed/race, pin it
  "It's green but the feature is broken"    Self-validating           replace println! with a real assertion
  "No test exists for this old behaviour"    Timely                    pin behaviour before it's a black box

A flaky, slow, or order-dependent test is never bad luck. It is a FIRST violation with a hidden cause, and the cause is findable. That reframing is the whole value of the checklist: it converts “our tests are unreliable” into a short list of specific, fixable defects.

The architect’s lens

Why does it exist? FIRST exists because a test suite’s real product is a fast, trustworthy feedback loop — and each property guards one way that loop silently rots (too slow to run, too coupled to trust, too flaky to believe, too vague to catch bugs, too late to be cheap).
What problem does it solve? It replaces the unfalsifiable complaint “our tests are bad” with a five-point diagnosis that names the defect and points at the fix.
What are the trade-offs? Isolation and repeatability cost setup (unique temp files, injected clocks, fakes); speed sometimes costs realism, which is exactly why integration tests like kvlite’s TCP suite deliberately trade speed for end-to-end truth in a separate tier.
When should I avoid it? Never wholesale — but apply it per tier. An integration or end-to-end test legitimately relaxes Fast (it uses real I/O by design); it must never relax Isolated, Repeatable, or Self-validating.
What breaks if I remove it? Drop Fast and the suite stops running on save. Drop Isolated and green becomes order-dependent. Drop Repeatable and red stops meaning “broken.” Drop Self-validating and the suite passes while the code is broken. Drop Timely and the tests that mattered are never written.

Check your understanding

Why is “Fast” ultimately a statement about how often the test runs, rather than about the test in isolation? Use the kvlite TCP suite to illustrate the trade-off.
In temp_wal(), two things make the path unique — the process id and an atomic counter. What distinct isolation problem does each one solve, and what would break if you dropped either?
Name the three classic sources of non-repeatability, and for each give the general fix.
A colleague says a test is “flaky — just re-run it.” Reframe that using FIRST. What is the diagnosis, and what is the actual next step?
A test ends with println!("{result}") and no assertion. Which FIRST property does it violate, why is it dangerous specifically because it is green, and how do you fix it?

Show answers

A 200 ms suite runs on every save, dozens of times an hour, as an unnoticed reflex; a two-minute suite becomes a decision people increasingly skip. So speed determines whether the test protects you at all. The kvlite TCP suite binds a real socket and spawns a thread — slower by design, because its job is end-to-end wire truth. That belongs in a separate, slower tier; your in-memory unit tests must stay fast.
The process id separates one cargo test run (or a stale file from a previous run) from another, so concurrent or leftover runs can’t collide. The atomic counter hands each call within a run a fresh integer, so two tests in the same run can’t collide. Drop the pid and separate runs clash; drop the counter and tests within one run clash. Either way you lose isolation and get order-dependent flakiness.
Wall-clock time → inject a fixed clock/instant. The network → replace the real dependency with a seam (stub/fake) so the test doesn’t reach the internet. Randomness/ordering (unseeded RNG, hash-map order, thread races) → seed it, sort it, or remove the race so the input is fixed.
Diagnosis: “flaky” is a Repeatable violation — the result changes without the code changing, which means a hidden input (clock, network, seed, or race) is leaking in. The next step is not “re-run”; it is to find and pin that hidden input. Flaky is a diagnosis, not bad luck.
It violates Self-validating — it prints instead of asserting, so no machine ever checks the verdict. It’s dangerous because it’s green: it stays green even when the code is broken, since nothing compares the output to an expectation. Fix it with a real assertion (assert_eq!, matches!, assert!) the runner can enforce without a human in the loop.