System & End-to-End Tests — The Whole Thing

Integration tests let two or three of your units meet and checked that they agreed on the contract between them. We climbed one rung: still in-process, still fast, but no longer a single function in a vacuum. Now we climb to the top rung, where the abstractions run out.

At this level we stop poking at the inside of the program. We assemble the whole thing — every module, the real I/O, the real argument parser, the real exit code — and we drive it the way a user actually drives it: from the outside, through the front door. This is where confidence is highest, because there is nothing left to mock. It is also where cost and flakiness are highest, because there is nothing left to mock.

System testing vs end-to-end testing

These two terms get used interchangeably, and for a small CLI they nearly collapse into one. But they answer different questions, so it is worth keeping them apart.

System testing asks: does the fully assembled system behave correctly in a realistic environment? You take the whole binary — or the whole service, with its real database, its real config file, its real file system — and you exercise its behaviour. The emphasis is on assembled and realistic: no stubs standing in for the disk, no in-memory fake for the store. Whatever production uses, this uses.
End-to-end (E2E) testing asks: does a complete user journey work, start to finish, through the real entry point? The emphasis is on the path. You do not call a function; you do what the user does — you run the command in a shell, you click through the web form, you send the HTTP request a real client would send — and you check what the user would see at the end.

   what each level actually touches

   unit         [ one function ]
   integration  [ module ⇄ module ]
   system       [ ── the whole assembled binary ── ]   realistic env
   end-to-end   $ mytool --flag file.log            ▶  as the shell runs it
                └ real argv → real stdout/stderr → real exit code ┘

The distinction is one of emphasis, not of a hard wall. A system test can be end-to-end; an E2E test is always exercising the system. What matters is the shared property that sets this whole tier apart from everything below it: nothing is substituted. The thing under test is the thing you ship.

Why this gives the highest-fidelity confidence

Every level below this one buys speed by replacing something real with something fake or partial. A unit test replaces collaborators with stubs. An integration test might replace the disk with a temp file or the network with a loopback socket. Each substitution is a small lie you tell for speed — usually a harmless one, but a lie nonetheless.

An end-to-end test tells no lies. Consider what actually runs when a shell invokes a CLI:

   $ logwise app.log --level error --strict
        │
        ├─ the OS parses argv and hands it to the process
        ├─ clap parses the flags (--level, --strict) — real parsing, real errors
        ├─ the file is opened and read from the real file system
        ├─ each line runs through the real parse + error chain
        ├─ output is written to real stdout / stderr
        └─ the process exits with a real status code the shell can see

A unit test of the line parser never touches clap, never opens a file, never sets an exit code. An E2E test touches all of it — and every one of those is a place a real user hits a real bug. Here is the actual E2E harness for the logwise CLI in this repo (rust/logwise/tests/cli.rs):

use std::process::Command;

/// Run the *built binary* with real args; capture what the shell would see.
fn run(args: &[&str]) -> (bool, String, String) {
    let output = Command::new(env!("CARGO_BIN_EXE_logwise"))
        .args(args)
        .output()
        .expect("failed to launch the logwise binary");
    (
        output.status.success(),                              // real exit code
        String::from_utf8_lossy(&output.stdout).into_owned(), // real stdout
        String::from_utf8_lossy(&output.stderr).into_owned(), // real stderr
    )
}

#[test]
fn strict_mode_fails_on_the_malformed_line() {
    let (ok, _, stderr) = run(&["samples/app.log", "--strict"]);
    assert!(!ok, "strict mode should exit non-zero on a bad line");
    assert!(stderr.contains("could not parse line 15"));
}

#[test]
fn missing_file_reports_an_error() {
    let (ok, _, stderr) = run(&["does-not-exist.log"]);
    assert!(!ok, "a missing file should be a fatal error");
    assert!(stderr.contains("could not read log file"));
}

Notice what this test cannot be fooled about. It does not assert that some function returns an Err; it asserts that the process exits non-zero — the thing a shell script or a CI pipeline actually branches on. It does not assert that a formatter produced a string; it asserts the string appeared on stderr, where a user would read it. If someone accidentally prints the error to stdout, or forgets to set the exit code, or wires --strict to the wrong flag, every unit test still passes green and this test is the one that catches it. That is the fidelity you are buying: the test’s definition of “correct” is the same as the user’s.

A single logwise E2E assertion covers a surprising amount of surface at once.

   one invocation:  logwise app.log --level error --strict

   code paths touched in a single run
   ───────────────────────────────────────────
   argv → clap parse ............  ~1 layer
   flag validation (--level enum)   ~1 layer
   file open + read ............... ~1 layer
   per-line parse + error chain ... N lines
   histogram + summary format ....  ~1 layer
   stdout/stderr write ...........  ~1 layer
   exit-code selection ...........  ~1 layer
   ───────────────────────────────────────────
   ≈ the entire program, in ~5–20 ms per test

   compare: a unit test of the parser exercises
   ONE of those layers in ~5–50 µs

Roughly three orders of magnitude slower per test — and it exercises the whole stack. That trade is exactly why you want many unit tests and few of these.

The costs, stated plainly

Nothing is free. The same property that makes these tests trustworthy makes them expensive.

They are the slowest to run

An E2E test pays for the real work: process spawn, real disk I/O, real parsing, sometimes a real network round-trip or a real database. A microsecond-scale unit test becomes a millisecond-scale system test — and if a browser or a container is involved, a second-scale one. A suite of ten thousand unit tests can finish before a hundred E2E tests do. That cost compounds every single time anyone runs the suite, forever.

They are the most expensive to maintain

An E2E test is coupled to the whole system’s observable behaviour, so it breaks whenever any of that changes — a reworded error message, a renamed flag, a new required argument, a different output format. Much of that churn is not a real regression; it is the test noticing a deliberate change. Every one of those still costs a human a diagnosis and an edit. The broader the test’s reach, the more often something in its reach moves.

They are the prime source of flaky tests

A flaky test is one that passes and fails without any change to the code under test — green on one run, red on the next, green again on a retry. Flakiness lives almost entirely in this top tier, because this tier touches all the non-deterministic parts of the world that the lower tiers mocked away:

   where flakiness comes from (all live up here)

   ⏱  timing / races ...... "the server wasn't ready yet"
   🌐  the network ......... a real socket, a real timeout
   💾  shared state ........ a leftover temp file, a port in use
   🔀  ordering ............ test B depends on test A's side effects
   🕰  time & clocks ....... "expires tomorrow" fails at midnight
   🎲  unstable ordering ... a HashMap iterated in random order

Notice how the kvlite server test in this repo defends against two of these on purpose: it binds to port 0 so the OS hands it a free port (no “address already in use” collision), and it uses a unique temp WAL file per test (no leftover state from a previous run). That care is the price of admission at this level — you write the same amount of code to prevent flakiness as you do to test the behaviour.

Why flakiness is corrosive

This is the part that matters most, and it is a human problem, not a technical one.

A test suite’s entire value rests on one belief: red means broken. The moment that belief cracks — the moment a red result might just be “that flaky one again, hit retry” — the suite starts to die. People re-run failed jobs on reflex. They merge over red with a shrug. And crucially, they stop being able to tell a flaky red from a real red, because both look identical. A genuine regression now hides in plain sight, wearing the same colour as the noise everyone has learned to ignore.

A test suite people have learned to ignore is worse than no suite at all. No suite is honest about giving you no safety. A flaky suite lies: it looks like safety, it costs real time to run, and it trains your team to disbelieve the one signal that was supposed to save them.

So flakiness is not a minor annoyance to be tolerated. A flaky E2E test is a defect in the test itself, and it deserves the same urgency as a defect in production code: fix the race, pin the clock, isolate the state — or delete the test. Never let it linger, because it poisons every honest test around it.

On 1 August 2012, the trading firm Knight Capital deployed new code to its automated order router. A deployment reportedly reached only some of the servers, leaving one running old logic that reacted to a repurposed flag. Within roughly 45 minutes the system fired millions of unintended orders into the market, and the firm lost on the order of $440 million — enough to nearly destroy the company overnight.

The details are debated, but the testing lesson is not: the failure lived in the fully assembled, deployed system under real conditions — exactly the seam that only a system/end-to-end test can reach. No unit test of any single component would have seen it, because every component may well have been individually correct. It was the whole thing, running the way it really runs, on real infrastructure, that was wrong. That is precisely the confidence this tier exists to buy — and precisely why you cannot skip it entirely, however tempting its cost makes that.

Keep them few and high-value

The lesson of cost and flakiness is not “avoid E2E tests.” It is “spend them deliberately.” Because each one is slow, brittle, and a flakiness risk, you want the fewest tests that still cover what genuinely matters: the critical user journeys.

Ask, for each candidate E2E test: if this exact path broke in production, would it be an incident? If yes, it earns a place at the top of the pyramid. If it is just another permutation of inputs, push it down — test that permutation at the unit level, where it is cheap and stable, and let the E2E test prove only that the journey holds together.

   DO cover (few, load-bearing):        DON'T cover here (push down):
   ─────────────────────────────        ─────────────────────────────
   ✓ happy path start→finish            ✗ every flag combination
   ✓ the primary error path             ✗ every malformed-input variant
     (missing file → exit ≠ 0)          ✗ every boundary value
   ✓ the one flag that changes          ✗ formatting edge cases
     behaviour the most (--strict)        (test those as units)

For logwise, that is a handful of tests: it summarizes a real file (happy path), it respects --strict by exiting non-zero (the behaviour-defining flag), and it fails cleanly on a missing file (the primary error path). Every other case — each log level, each parse edge — is checked far more cheaply one level down. A few high-value E2E tests, resting on a broad base of fast unit tests: that shape is the whole point, and the next pages make it explicit.

The architect’s lens

Why does it exist? Because every lower level buys speed by substituting something real with a fake, and those substitutions can hide bugs that only appear when the whole thing runs for real — in the argument parser, the I/O, the error chain, the exit code.
What problem does it solve? It closes the gap between “all the parts pass their own tests” and “the assembled product does what the user needs,” by exercising the system through the user’s actual entry point with nothing mocked.
What are the trade-offs? Maximum confidence in exchange for maximum cost: slowest to run, most expensive to maintain, and the prime home of flakiness. Every E2E test is a standing liability as well as an asset.
When should I avoid it? When a cheaper level can prove the same thing. Do not test input permutations, boundary values, or formatting quirks here — push them down to unit tests and reserve this tier for critical journeys.
What breaks if I remove it? You lose the only signal that the deployed, assembled system works the way it ships — integration seams, exit codes, and the real user path go unverified, which is exactly the class of failure that becomes a Knight-Capital-shaped incident.

Check your understanding

Both a system test and an E2E test avoid mocking. What is the difference in emphasis between them?
A unit test of logwise’s line parser passes, but a user reports that --strict never fails the command. Why would only an E2E test catch this, and what specific thing does the E2E test assert that the unit test cannot?
Give three concrete sources of flakiness, and explain why all three live at this level and not at the unit level.
“A flaky test suite is worse than no test suite.” Explain the reasoning behind this claim in terms of what a test suite’s value depends on.
You are deciding whether to add an E2E test for a new input-validation edge case. What question do you ask, and where does the test most likely belong?

Show answers

System testing emphasizes the assembled system in a realistic environment — the whole binary with its real I/O and config, checked for correct behaviour. E2E testing emphasizes the complete user journey through the real entry point — doing what the user does, start to finish, and checking what the user would see. For a small CLI they nearly coincide; the shared, defining property is that nothing is substituted.
Because the bug is not in the parser at all — it is in how the assembled program wires the --strict flag to the process exit code, which the unit test never touches. The E2E test asserts on the real exit status (output.status.success() is false) and on stderr, the things a shell actually branches on. A unit test can only see a return value inside the process.
Any three of: timing/races (a server not yet ready), the real network (sockets, timeouts), shared state (leftover temp files, a port already in use), test ordering (test B depending on test A’s side effects), clocks (“expires tomorrow” failing at midnight), and unstable iteration order (a randomly-ordered map). They live here because this is the only tier that touches the real, non-deterministic world; lower tiers mocked all of it away.
A suite’s value rests on the belief that red means broken. Flakiness breaks that belief: people learn to re-run and merge over red, and a genuine regression becomes indistinguishable from noise everyone ignores. No suite is at least honest about offering no safety; a flaky one lies — it looks like safety, costs real time, and trains the team to disbelieve the one signal meant to protect them.
Ask: if this exact path broke in production, would it be an incident? An input-validation edge case almost never is on its own, so it belongs at the unit level, where it is cheap and stable. Reserve the E2E tier for the critical journeys — the happy path, the primary error path, and behaviour-defining flags.