Frameworks and Runners as Concepts

The pyramid told you where your tests should live and in what proportion. Before you write the first one, you need to know what the machinery underneath is doing on your behalf — because you will spend years using it, and most people never look past the tool’s name.

This page is deliberately tool-free for as long as possible. “JUnit”, “pytest”, “Jest”, “Go’s testing”, “Rust’s #[test]” — these are brands. Underneath every one of them are the same two roles and the same universal shape. Learn the concepts once and every framework you ever meet becomes a matter of syntax, not substance.

Two roles, one bundle

When people say “a testing framework” they usually mean one download that quietly does two different jobs. Separating them is the whole point of this page.

  YOUR TEST CODE                         THE RUNNER
  ─────────────                          ──────────
  the FRAMEWORK gives you:               the RUNNER gives you:
    • a way to declare "this is           • DISCOVERY  — find all the tests
      a test"                             • EXECUTION  — call each one
    • assertions (assert_eq!, expect())   • ISOLATION  — keep them from
    • setup / teardown hooks                colliding
    • fixtures for shared state           • REPORTING  — pass/fail, timing,
                                            machine-readable output

• The framework is a vocabulary. It gives you the words to express a test: a way to mark a function as a test, a set of assertions to state expectations, and hooks to arrange and clean up shared state. It is a library you call.
• The runner is an engine. It finds everything you marked, runs it, keeps the runs from interfering with each other, and tells you and your CI what happened. It is a program that calls you.

Some tools ship both in one binary (Rust’s cargo test, Go’s go test); some split them (pytest is a runner that discovers assert-based tests; JUnit is a framework the Gradle/Maven runner drives). The split is conceptual regardless of packaging — and once you see it, “why did my test not run?” and “why is this test flaky?” become questions about different halves of the system.

The universal shape: arrange–act–assert

Every good test, in every language, has the same three beats. Naming them is not pedantry — it is how you keep a test readable and how you spot when a test is doing too much.

  ARRANGE   set up the world: inputs, dependencies, starting state
  ACT       perform the ONE action under test
  ASSERT    state what must now be true

Here is that shape in a real Rust integration test (from kvlite/tests/), driving a key-value server over a real socket:

#[test]
fn tcp_set_get_del_roundtrip() {
    // ARRANGE — a fresh database on a throwaway WAL file, server on a free port
    let path = temp_wal();
    let (db, _) = Db::open(&path).unwrap();
    let listener = TcpListener::bind("127.0.0.1:0").unwrap();
    let addr = listener.local_addr().unwrap();
    thread::spawn(move || { let _ = server::serve_on(listener, db); });
    let stream = TcpStream::connect(addr).unwrap();
    let mut writer = stream.try_clone().unwrap();
    let mut reader = BufReader::new(stream);

    // ACT + ASSERT — one exchange per line, expectation stated inline
    assert_eq!(round_trip(&mut reader, &mut writer, "PING"), "PONG");
    assert_eq!(round_trip(&mut reader, &mut writer, "GET name"), "NIL");
    assert_eq!(round_trip(&mut reader, &mut writer, "SET name ada"), "OK");
    assert_eq!(round_trip(&mut reader, &mut writer, "GET name"), "VALUE ada");
}

#[test] is the framework marking a function as a test. assert_eq! is the framework providing an assertion. cargo test is the runner that found this function, ran it, and reported it. Nothing here is special to Rust — swap the syntax and it is a pytest, a JUnit, a Go test.

Assertions: the load-bearing line

An assertion is the only line that can fail a test. Everything else is scaffolding. A good assertion says one specific true-or-false thing, and when it fails, the framework prints what you expected vs. what you got — the diff is the whole value.

  assert_eq!(got, "PONG")   →  on failure prints:  left: "PANG"  right: "PONG"

A test with no assertion is not a test — it is a program that happens to run. It passes as long as the code doesn’t crash, which means it will keep passing long after the behaviour it “checks” has quietly broken.

Frameworks tend to offer two flavours of assertion, and knowing the difference saves you a lot of grief:

  HARD assertion   →  fails the test AND stops it immediately
                      (the rest of the test body never runs)
  SOFT assertion   →  records the failure but keeps going, so you
                      see EVERY mismatch in one run, not just the first

Most language built-ins (assert_eq!, assert) are hard: the first mismatch aborts. That is usually what you want, because after arrange has drifted, later assertions are lying anyway. Soft assertions earn their keep when you are checking many independent fields of one result and want the full picture in a single run. Reach for soft assertions sparingly — a test that reports ten failures at once is often a test that should have been ten tests.

Setup, teardown, and fixtures

The temp_wal() helper and Db::open above are arrange. When many tests need the same arrangement, you lift it into setup (runs before each test), teardown (runs after, to clean up), or a fixture — a named, reusable piece of prepared state the runner hands to each test that asks for it.

  setup      → build the shared world (a temp dir, a DB connection, a fake clock)
  fixture    → a named handle to that world, injected into each test
  teardown   → tear it down so the next test starts clean

The rule that makes fixtures safe: each test gets its own instance, or a freshly reset one. The moment two tests share the same mutable fixture, you have re-introduced the exact problem isolation exists to prevent.

Isolation and independence

This is the concept most teams learn the hard way.

A test must pass or fail on its own merits, regardless of what ran before it, after it, or beside it.

Two properties follow, and both are non-negotiable:

• Order independence. If your suite passes in order 1-2-3 but fails as 3-2-1, that is a bug in the tests, not a fluke. It means test 1 left state behind that test 2 silently depended on. Good runners let you shuffle test order precisely to smoke this out.
• No shared mutable state. A global counter, a row in a shared database, a file on disk with a fixed name, a real wall-clock — any of these, mutated by one test and read by another, makes the suite flaky: it passes and fails without the code changing. Notice the kvlite test above binds to port 0 (OS picks a free port) and writes to a per-process, per-call WAL filename. That is isolation engineered in, not hoped for.

  ORDER-DEPENDENT (a bug)                 ISOLATED (correct)
  ───────────────────────                 ──────────────────
  test_a: creates user "bob"              each test: creates its OWN user
  test_b: assumes "bob" exists  ✗         each test: unique port / temp file
      → passes only if a runs first           → passes in ANY order, in parallel

Isolation is also what buys you parallelism. A runner can only run tests concurrently if they don’t touch each other’s state — so the discipline that kills flakiness is the same discipline that makes your suite fast. The two goals are one goal.

A practical checklist for whether a test is truly isolated:

• Does it create its own inputs, or read something a previous test wrote?
• Does it use a unique resource name (temp file, port, table, key prefix), or a fixed one that a sibling could also grab?
• Does it depend on the wall clock, the network, or a real third-party service — any of which can change under it between runs?
• Does it clean up after itself, so the next test — and the ten-thousandth run of this test — starts from the same blank slate?

If the answer to any of the first three is “yes,” you have a latent flake. The runner cannot fix this for you; isolation is a property of how the test is written, and the runner’s shuffle-and-parallelise switches merely reveal whether you got it right.

It actually happened — the shared-state flake

A recurring, un-dated pattern across many real CI systems: a test suite is green for months, then starts failing “randomly” only on the build server, never on a laptop. The cause is almost always the same — two tests share mutable state (a database row, a temp file, a static singleton, a fixed port) and the CI machine runs them in a different order or in parallel where the laptop ran them serially. The lesson is blunt: a test that only passes in one order was never really passing. Fix the isolation, not the runner’s parallelism setting.

Reporting: the output is an interface

The last job of the runner is to tell someone what happened — and “someone” is both you and a machine.

• For a human: pass/fail counts, which tests failed, the assertion diff, and timing so you can see the slow tests dragging the suite.
• For a machine: a structured result file — JUnit XML and TAP are the long-standing common formats — that a CI system parses to decide whether to go red, and to annotate the failing line on a pull request.

  running 3 tests
  test tcp_set_get_del_roundtrip ... ok      (0.04s)
  test wal_survives_restart ......... ok      (0.11s)
  test rejects_bad_command .......... FAILED  (0.01s)

  test result: FAILED. 2 passed; 1 failed; finished in 0.16s
       ↓ same run, machine-readable
  <testsuite tests="3" failures="1" time="0.16"> ... </testsuite>

A test result no machine can read is a test result that cannot gate a merge — so the reporter, unglamorous as it is, is the bridge between “the tests ran” and “the pipeline is trustworthy.”

Notice the timing column is not decoration. It is your only early-warning system for a suite going slow. A suite that takes four seconds today and forty seconds in six months got there one 200-millisecond test at a time, and the per-test timing is the only place that creep is visible before it becomes a merge-blocking tax on every engineer.

By the numbers — what an assertion diff saves you

The value of a framework’s assertion diff is measured in the debugging time it removes. Compare the two failure messages a runner can print for the same broken test:

  Without a diff:   "assertion failed"           → you now re-run, add prints, guess
  With a diff:      left:  "PANG"                 → you already know: an off-by-one
                    right: "PONG"                    typo in the response string

  Rough time-to-cause, same bug:
    no diff    →  edit test, add debug output, re-compile, re-run   ~2–5 min
    good diff  →  read the two lines, fix                           ~15 sec

Multiply the difference by the number of test failures an active team sees in a week — easily dozens — and the assertion diff is quietly one of the highest-leverage features the framework ships. Treat a failure message that only says “failed” as a bug in the test, not just bad luck.

Under the hood — how discovery finds your tests

Discovery is not magic; it is convention plus reflection. A runner uses one of three strategies, and knowing which explains most “my test didn’t run” mysteries:

• Attribute / annotation — you tag the function (#[test], @Test, @pytest.mark), and the runner (or compiler) collects everything tagged. Rust’s cargo test compiles a special test harness that links only functions marked #[test] inside #[cfg(test)] modules, plus every file in the tests/ directory as its own integration crate.
• Naming convention — the runner globs for files or functions matching a pattern (test_*.py, *_test.go, *.spec.js). Name it wrong and it is invisible; there is no error, it simply never runs.
• Registration — you explicitly add tests to a suite object. Rare in modern tools, common in older ones.

Rust makes the framework/runner split unusually visible: unit tests live beside the code in #[cfg(test)] mod tests (compiled out of release builds entirely), while integration tests live in a separate tests/ directory and can only touch the crate’s public API — the compiler enforces the same isolation boundary a real user would hit.

  src/lib.rs
    fn parse(..) { .. }
    #[cfg(test)]                 ← compiled ONLY under `cargo test`;
    mod tests {                     absent from the shipped binary
      #[test] fn parses_ok() { assert_eq!(parse("SET a b"), ..); }
    }                            ← UNIT: can see private internals

  tests/server.rs               ← INTEGRATION: its own crate,
    #[test] fn tcp_roundtrip()      sees only the PUBLIC API,
      { .. }                        exactly like a real caller

That layout is a design decision, not an accident: it forces you to test private helpers from inside the module and public behaviour from outside it, so your integration tests can never quietly reach past the front door. Any framework in any language can be used this way — Rust simply makes the boundary a compiler rule rather than a convention you have to remember.

The architect’s lens

• Why does it exist? To turn “does the code work?” from a manual, remembered ritual into a repeatable, automated fact — and to give that fact a shape (arrange–act–assert) and an audience (humans and CI).
• What problem does it solve? Writing test logic is only half the job; finding, running, isolating, and reporting hundreds of tests reliably is the other half. The framework standardises the first, the runner industrialises the second.
• What are the trade-offs? A framework imposes structure and vocabulary you must learn, and its assertions and fixtures can hide real complexity behind convenient magic. Fixtures that share state trade convenience for flakiness if you’re careless.
• When should I avoid it? Almost never — but don’t reach for a heavyweight framework to check a five-line pure function when the language’s built-in one already does it. And don’t let the framework’s features tempt you into order-dependent, shared-state tests.
• What breaks if I remove it? Discovery, isolation, and machine-readable reporting all become manual. Without them you can still write tests, but you cannot trust that they all ran, that they didn’t poison each other, or that CI can gate a merge on them.

Check your understanding

1. In one sentence each, state the distinct job of a framework and a runner.
2. Name the three beats of the universal test shape, and say which one is the only line that can fail the test.
3. A suite passes in order A-B-C but fails as C-B-A. Is this a flaky runner or a bug in the tests? What is the underlying cause?
4. Why is the discipline that kills flakiness the same discipline that lets a runner run tests in parallel?
5. Why does a runner emit a machine-readable report (e.g. JUnit XML) in addition to human-readable output?

Show answers

1. The framework gives you the vocabulary to express a test — a way to mark a test, assertions, and setup/teardown/fixtures. The runner finds, executes, isolates, and reports those tests.
2. Arrange, Act, Assert. The assert is the only line that can fail the test; without an assertion the “test” merely runs and always passes.
3. A bug in the tests, not the runner. Some test left mutable state behind (a row, a file, a global) that a later test silently depended on — order independence has been violated.
4. Both require the same thing: no shared mutable state. Tests that don’t touch each other’s state can be reordered and run concurrently without interfering — so isolation delivers both correctness and speed at once.
5. Because a machine (the CI system) must parse the result to decide whether to go red and to annotate the failing line on a pull request; human-readable text alone cannot gate a merge.