Skip to content
Software Testing · First Principles

Tests as Living Documentation

The previous page cut a unit free of its slow, flaky neighbours by inserting seams. Once a unit is isolated, its tests become fast, deterministic, and — the point of this page — readable. A test that runs in a millisecond and never lies about its subject is not just a safety net. It is documentation.

Every other form of documentation shares one fatal flaw: nothing forces it to stay true. A comment, a README, a wiki page, a doc-comment above a function — all of them are prose written next to code, and prose does not run. The code drifts, the prose stays, and now the documentation is a lie that no compiler, no CI job, no reviewer is obligated to catch. A test is different. A test is a claim about behaviour that is executed. When the behaviour drifts away from the claim, the test goes red. Documentation that fails when it lies is the only documentation you can trust.

Why prose rots and tests don’t

Section titled “Why prose rots and tests don’t”

Consider the two ways to record “GET on a missing key returns NIL.”

As a comment:                          As a test:


// GET returns NIL for a missing key   assert_eq!(get("name"), "NIL");
                                        // ↑ runs every CI build


Someone changes NIL → NOT_SET.         Someone changes NIL → NOT_SET.
The comment still says NIL.            The test goes RED immediately.
Nothing fails. The lie ships.          The lie cannot ship until the
                                        claim is updated to match.

The comment and the assertion say the same thing. The difference is entirely in who is responsible for keeping them true. The comment relies on a human noticing, remembering, and caring during an unrelated change — the weakest link in software. The assertion relies on the build, which checks every claim on every push, forever, without being asked. This is the whole argument: a test is an executable specification. It does not describe behaviour; it pins it. Break the behaviour and you break the build, which is exactly the feedback prose can never give you.

This reframes what a test is for. A test’s first job is to catch regressions. Its second, almost-free job is to answer a new reader’s question — how does this thing actually behave? — with an answer that is guaranteed current, because a stale answer would be a failing build.

Read one test, learn the whole protocol

Section titled “Read one test, learn the whole protocol”

Here is the payoff, made concrete. Below is a single real integration test for kvlite, a tiny key–value server that speaks a line-based protocol over TCP. Imagine you have never seen kvlite, there is no README, and this test is all you have. Read it as documentation.

#[test]
fn tcp_set_get_del_roundtrip() {
    // ...bind a server on a random port, connect a client...


    assert_eq!(round_trip(&mut reader, &mut writer, "PING"), "PONG");
    assert_eq!(round_trip(&mut reader, &mut writer, "GET name"), "NIL");
    assert_eq!(
        round_trip(&mut reader, &mut writer, "SET name ada lovelace"),
        "OK"
    );
    assert_eq!(
        round_trip(&mut reader, &mut writer, "GET name"),
        "VALUE ada lovelace" // value with spaces survives the wire
    );
    assert_eq!(round_trip(&mut reader, &mut writer, "DEL name"), "DELETED");
    assert_eq!(round_trip(&mut reader, &mut writer, "DEL name"), "NOT_FOUND");
    assert!(round_trip(&mut reader, &mut writer, "bogus").starts_with("ERR"));
}

Without a word of prose, you now know the entire contract:

PING                 →  PONG            liveness check
GET <key>  (missing) →  NIL             absence has a distinct reply, not ""
SET <key> <value>    →  OK              writes acknowledge
GET <key>  (present) →  VALUE <value>   reads echo the stored value
DEL <key>  (present) →  DELETED         deletes confirm they removed something
DEL <key>  (missing) →  NOT_FOUND       deleting nothing is reported, not silent
<garbage>            →  ERR ...         unknown commands fail loudly with a prefix

You learned that the server is line-oriented, that every command gets exactly one line back, that “not found” and “error” are different outcomes (a missing key is NIL on GET but NOT_FOUND on DEL — a distinction a paragraph of prose would probably blur), and that malformed input is rejected with a machine-readable ERR prefix rather than a crash or silence. That is a protocol specification, and it cannot be out of date, because if the server stopped saying PONG this test would fail before it ever reached you.

The test taught you things a README would omit

Section titled “The test taught you things a README would omit”

Notice what the test made unmissable that a hand-written doc would likely gloss:

  • NIL vs NOT_FOUND — two words for “absence,” used deliberately in two different places. A prose writer, summarising, tends to collapse these into one. The test keeps them apart because the code does.
  • bogus → starts_with("ERR"). The test documents the shape of the error contract (an ERR prefix) without over-committing to the exact message, so the message can improve without breaking the doc.
  • VALUE ada lovelace — a two-word value round-trips intact. That single assertion silently documents that the value field is the rest of the line, not a single token. We will return to why that specific line matters.

Structure a test so it reads like an example

Section titled “Structure a test so it reads like an example”

A test only documents well if it is legible. Two habits do most of the work: a name that states the behaviour, and an Arrange–Act–Assert body that reads like a worked example.

Name the behaviour, not the method. A test name is a one-line spec. test_get() documents nothing; get_missing_key_returns_nil() documents a rule. Read a good suite’s test names top to bottom and you get a table of contents for the unit’s behaviour:

get_missing_key_returns_nil
set_then_get_returns_stored_value
value_with_spaces_survives_the_wire
del_present_key_returns_deleted
del_absent_key_returns_not_found
unknown_command_returns_err_prefix

That list is the specification. Someone who never opens the bodies already knows what kvlite promises.

Use AAA so each test is one documented example of one behaviour. As covered in What Is a Unit?, Arrange–Act–Assert gives a test three visually distinct blocks: set up the situation, perform the one action, check the one result. When each test isolates a single behaviour, its body reads like a captioned example in a manual — given this, when I do that, I get this.

def test_del_absent_key_returns_not_found():
    # Arrange — a store that has never seen this key
    store = KvStore()


    # Act — delete the key that isn't there
    reply = store.handle("DEL ghost")


    # Assert — absence on delete is reported, not silently swallowed
    assert reply == "NOT_FOUND"

A reader scanning that test learns one rule cleanly: deleting a key that does not exist is reported, not silently ignored. Cram three behaviours into one test and the documentation blurs — you can no longer point at a single test and say “this is the DEL-absent rule.” One behaviour per test keeps each example about exactly one thing, and keeps the failure message precise when it breaks.

Encode the corners as named tests

Section titled “Encode the corners as named tests”

Prose documentation almost never covers edge cases — it describes the happy path and trails off. A test suite documents the corners by construction, because each corner is a named test that must pass. Two lines from kvlite’s roundtrip are edge cases masquerading as ordinary assertions, and they are worth pulling out into their own named tests precisely so the docs cover the corners.

A value with spaces survives the wire.

def test_value_with_spaces_survives_the_wire():
    # Arrange
    store = KvStore()


    # Act — the value is "ada lovelace", a value containing a space
    store.handle("SET name ada lovelace")
    reply = store.handle("GET name")


    # Assert — the whole value comes back, space intact
    assert reply == "VALUE ada lovelace"

This test documents a parsing decision that is easy to get wrong: the value is everything after the second token, not a single word. Without this test, a future refactor that splits the line on whitespace and stores only "ada" would pass every single-word test and quietly corrupt every multi-word value. The named test is the doc and the guardrail for that corner.

Deleting twice returns NOT_FOUND the second time.

def test_del_twice_second_is_not_found():
    # Arrange — a key that exists
    store = KvStore()
    store.handle("SET name ada")


    # Act — delete it, then delete it again
    first  = store.handle("DEL name")
    second = store.handle("DEL name")


    # Assert — DEL is not idempotent in its *reply*
    assert first  == "DELETED"
    assert second == "NOT_FOUND"

This documents that DEL distinguishes “I removed something” from “there was nothing to remove” — a subtlety a one-line description (“DEL removes a key”) would erase entirely. The second DEL is the corner, and naming it makes the corner part of the permanent, executable spec.

The pattern generalises: for every edge case your unit handles, there should be a test whose name states the edge case. The suite then reads as a complete contract — happy path and corners — that is impossible to leave stale.

The failure modes: tests that document the wrong thing

Section titled “The failure modes: tests that document the wrong thing”

Not every test documents behaviour. Two common mistakes produce tests that look like documentation but teach a reader the wrong lesson — or nothing at all.

Tautological tests document nothing. A tautological (or “self-fulfilling”) test is one whose assertion is guaranteed true by construction, so it can never fail and therefore records no real claim.

def test_discount():
    price = 100
    expected = price * 0.9      # the test recomputes the code's own logic
    assert apply_discount(price) == expected

If apply_discount is price * 0.9, this test asserts x == x. It is green forever, it survives any bug that changes the discount rate on both sides, and it documents only “the function does whatever it does.” A reader learns nothing about what the discount is. The fix is to assert a concrete, independently-known expected value — assert apply_discount(100) == 90 — which pins the behaviour and tells the reader the rule is “10% off.” Good documentation states the answer; a tautology restates the question.

Over-mocked tests document the implementation, not the behaviour. When a test replaces so many collaborators with mocks that its assertions are all “and then it called repo.save() with these arguments,” it stops documenting what the unit does and starts documenting how the unit is currently wired.

def test_register_user():
    repo = Mock()
    register_user("ada", repo)
    repo.insert.assert_called_once_with("ada")   # implementation detail

This test passes only while register_user happens to call a method named insert with exactly that argument shape. Rename the method, batch the write, add a cache — all behaviour-preserving — and the test breaks, teaching the next reader that the plumbing is the contract. It also cannot catch a real bug: insert might throw, or store the wrong thing, and the mock cheerfully records the call anyway. Recall from Test Doubles and Isolating Dependencies that mocks are for unavoidable dependencies (the network, the clock), not for every collaborator. A test that documents behaviour asserts on the observable result — the row exists, the reply is OK — not on which private methods got called along the way.

def test_registered_user_can_be_fetched():
    store = KvStore()                     # a real, in-memory fake
    register_user("ada", store)
    assert store.get("ada") is not None   # observable behaviour, not plumbing

The heuristic: if a behaviour-preserving refactor breaks the test, the test was documenting the implementation. Documentation should survive refactors, because the contract survives refactors. That is the line between a test that teaches and a test that merely echoes today’s code.

The architect’s lens

Section titled “The architect’s lens”
  • Why does it exist? Because every non-executable form of documentation — comments, READMEs, wikis, doc-comments — can silently drift out of sync with the code, and nothing forces it back into truth. Tests are the one form of documentation the machine re-verifies on every build.
  • What problem does it solve? It kills stale documentation. A test that documents a behaviour fails the instant that behaviour changes, so the “docs” and the code can never quietly disagree — you learn about the drift as a red build, not as a confused reader six months later.
  • What are the trade-offs? Tests-as-docs are precise but narrow: they show what the unit does, not always why it exists or how the whole system fits together — you still need a little prose for the big picture. And the guarantee only holds if the tests assert real behaviour; a tautological or over-mocked suite documents lies just as confidently.
  • When should I avoid it? Don’t lean on a test as documentation of behaviour it does not actually assert. A test that only checks that a mock was called documents plumbing, not contract, and a test that recomputes the code’s own arithmetic documents nothing. Use prose (sparingly) for rationale and architecture that no assertion can capture.
  • What breaks if I remove it? You are back to prose-only docs: comments that lie the moment someone refactors, a README describing a protocol the server no longer speaks, and new readers who cannot trust a single word of it — so they read the source anyway, which is the situation documentation was supposed to prevent.

Check your understanding

Section titled “Check your understanding”
  1. State the core argument for why a test is more trustworthy documentation than a comment or README, in terms of what happens when the described behaviour changes.
  2. A reader with no README opens only tcp_set_get_del_roundtrip. Name three distinct facts about kvlite’s protocol they can learn from it, including one that a summarising prose doc would likely blur.
  3. Why does naming a test del_absent_key_returns_not_found (rather than test_del) and giving it a clean Arrange–Act–Assert body make it read as documentation? What does “one behaviour per test” buy the reader?
  4. Pick one of kvlite’s edge cases — the space-containing value or the double DEL — and explain what contract detail its named test documents that a one-line description would erase.
  5. Define a tautological test and an over-mocked test, and give the one-line heuristic that tells you a test is documenting the implementation instead of the behaviour.
Show answers
  1. A comment or README is prose that does not run, so when the code’s behaviour changes the prose stays put and silently becomes a lie that nothing is obligated to catch. A test is an executed claim: when the behaviour drifts away from what the test asserts, the test goes red and the build fails, so a stale test cannot ship undetected. Documentation that fails the moment it lies is the only kind you can trust to be current.
  2. Any three of: PING returns PONG (liveness); a GET on a missing key returns NIL (absence is a distinct reply, not ""); SET returns OK; GET on a present key returns VALUE <value>; a value with spaces (ada lovelace) round-trips intact (so the value is the rest of the line, not one token); DEL on a present key returns DELETED but on an absent key returns NOT_FOUND; unknown input returns a line starting with ERR. The detail prose would likely blur: NIL (GET-missing) vs NOT_FOUND (DEL-missing) are two different replies for “absence,” used deliberately in two places — a summary tends to collapse them into one.
  3. The name is a one-line spec: it states the rule (deleting an absent key returns NOT_FOUND) rather than the method, so a reader scanning test names gets a table of contents of behaviour without opening any bodies. The AAA body reads like a captioned worked example — given an empty store, when I DEL a missing key, I get NOT_FOUND. “One behaviour per test” means each test is a single, unambiguous example the reader can point at as the rule for that behaviour, and it keeps the failure message precise when the rule breaks.
  4. Space-containing value: the named test documents that the value field is everything after the second token, not a single word — so a refactor that split on whitespace and stored only the first word would corrupt every multi-word value while a one-line “SET stores a value” doc would give no hint the corner exists. Double DEL: it documents that DEL distinguishes “I removed something” (DELETED) from “there was nothing to remove” (NOT_FOUND) — a distinction a one-line “DEL removes a key” description erases entirely.
  5. A tautological test asserts something guaranteed true by construction (e.g. recomputing the code’s own price * 0.9 on both sides), so it can never fail and records no real claim — the fix is to assert a concrete, independently-known value like == 90. An over-mocked test replaces so many collaborators with mocks that it only asserts which methods were called with which arguments, documenting today’s wiring rather than the observable result. The heuristic: if a behaviour-preserving refactor breaks the test, the test was documenting the implementation, not the behaviour.