End-to-End Tests and Their Cost

The last page handed you a dial: real vs fake dependencies, turned per test to trade fidelity for speed and stability. This page is what happens when you turn that dial all the way to the right for every collaborator at once and drive the whole thing through its front door. That is an end-to-end (E2E) test, and it sits at the very top of the pyramid: the most convincing test you can write, and the most expensive.

The overview drew the general shape — few, slow, highest fidelity. This page makes it concrete: what an E2E test actually is, the two levels you can run it at, why it belongs at the top of the pyramid and nowhere else, and the one rule of thumb that keeps it from bankrupting your suite.

What an E2E test actually is

An end-to-end test exercises the fully-assembled system through its real entry point, and asserts only on user-visible behaviour — the output a real user or client would see. Nothing inside is reached into, stubbed, or inspected. You send the same input a customer would, and you check the same result they would.

That definition has three load-bearing parts:

Fully assembled. The real server, the real database, the real cache, the real queue — the same binaries and config you deploy, wired together the way production wires them. If anything on the critical path is faked, it is not end-to-end; it is a wide integration test wearing a costume.
Real entry point. You go in through the door a user uses: an HTTP request to the public API, or a browser clicking a button. Not a function call. Not an internal method. The door.
User-visible behaviour. You assert on what comes out the door — the HTTP status and body, the text on the page, the redirect that fires — not on a private field, a log line, or a row you peeked at directly. If a refactor changes the internals but the user sees the same thing, the test must still pass.

   A unit test asserts here:        An E2E test asserts here:

        ┌──────────┐                 user ──► ┌─────────────────────┐
   ───► │ function │ ───► return              │  API / browser      │ ──► response
        └──────────┘                 (real request)                       (what the
        (one component,                      │  real server        │       user sees)
         rest faked)                         │  real DB · cache    │
                                             │  real queue         │
                                             └─────────────────────┘
                                          nothing on the path is faked

The reason people write E2E tests despite the cost is simple and hard to argue with: when a green E2E test says the checkout works, the checkout works. Every other level makes an assumption — “given the DB behaves, given the payment client is correct.” The E2E test makes none. It is the closest a test can get to standing behind a real user and watching them succeed. That is why it is the most convincing test you can write.

Two levels: API-level and browser-level

“End-to-end” is not one thing. There are two places you can put the entry point, and the choice is a fidelity-versus-cost decision all over again.

API-level E2E

You drive the system through its network API — an HTTP client firing real requests at the real server, real dependencies behind it. You never open a browser. The real Rust test in rust/kvlite/tests/server.rs is exactly this shape: it binds a real TCP listener, spawns the actual server, connects a real client socket, and asserts on the bytes that come back over the wire — PING → PONG, then a SET/GET round-trip. Nothing is mocked, because the whole point is to test the assembled system through its front door, not any function inside it.

// Drive the fully-assembled server through its real entry point (the socket),
// and assert only on what the client sees come back over the wire.
let stream = TcpStream::connect(addr).unwrap();
let mut writer = stream.try_clone().unwrap();
let mut reader = BufReader::new(stream);

assert_eq!(round_trip(&mut reader, &mut writer, "PING"), "PONG");
assert_eq!(round_trip(&mut reader, &mut writer, "SET name ada"), "OK");
assert_eq!(round_trip(&mut reader, &mut writer, "GET name"), "VALUE ada");

For a headless service — an API with no UI — this is end-to-end. The user is another program, and the door it uses is the API. You get most of the fidelity of a full system test at a fraction of the cost, because there is no browser to drive.

Browser-level E2E

For anything with a user interface, the real entry point is a browser, and API-level tests miss a whole class of bugs that live only in the UI: a button wired to the wrong handler, a form that never submits, a client-side validation that blocks a valid input, a redirect that loops. To catch those you drive a real browser with a tool like Playwright, Cypress, or Selenium — programmatically clicking, typing, and reading the rendered page exactly as a person would.

   Playwright / Cypress / Selenium
        │  click "Buy"
        ▼
   [ real browser ] ──HTTP──► [ real API ] ──► [ real DB ]
        ▲                                          │
        └──────── rendered page ◄──────────────────┘
   assert: "Order confirmed #1234" is visible on screen

Browser-level E2E is the highest-fidelity test that exists — it exercises your JavaScript, your CSS-driven visibility, your routing, your API, and your database in one shot, the way a customer does. It is also the most expensive: you are booting a browser, waiting on network and rendering, and asserting against a UI that changes for reasons that have nothing to do with correctness. Every bit of fidelity you gain by driving the UI, you pay for in speed and brittleness.

	API-level E2E	Browser-level E2E
Entry point	HTTP / network API	Real rendered browser
Catches UI bugs	No	Yes
Typical runtime	10s–100s of ms	seconds each
Brittleness	moderate	high (selectors, timing, rendering)
Best for	headless services, API contracts	user journeys through a real UI

The two levels are not rivals; they are two rungs of the same ladder, and mature suites use both. API-level E2E is where you prove the system — server plus data stores plus queues — answers correctly through the network door, cheaply enough to run often. Browser-level E2E sits one rung above it, adding the UI layer that the API level structurally cannot see, at a cost that forces you to keep it to a handful of tests. A useful instinct: prove as much as you can at the API level, and reserve the browser for the bugs that only a browser can find. Every assertion you can honestly make against the API instead of the rendered page is one you can make faster and keep greener.

As of mid-2026, Playwright, Cypress, and Selenium are the widely-used browser-driving tools; the specific market leader shifts, so treat the technique — drive a real browser, assert on rendered behaviour — as the durable idea and the tool name as a detail that will age.

Order-of-magnitude figures, not benchmarks — the ratios are the point, and they hold across most stacks.

   Level              Typical runtime   Relative cost to run+maintain
   ----------------   ---------------   -----------------------------
   Unit                 ~1 ms                1×
   Integration          ~10–100 ms          10×
   API-level E2E        ~0.1–1 s           100×
   Browser E2E          ~1–10 s          1000×

   Suite budget example (aim: full suite < 10 min on CI):
     4,000 unit    × 1 ms   =   4 s
       400 integ   × 50 ms  =  20 s
        40 API-E2E × 500 ms =  20 s
         8 browser × 5 s    =  40 s
     ------------------------------------
     total ≈ 84 s  →  4,000 unit tests cost less than 8 browser tests

The lesson in the arithmetic: you can afford thousands of unit tests or a few dozen E2E tests for the same time budget. That ratio is why the pyramid is shaped the way it is — it is an economic constraint, not a style preference.

Why E2E tests live at the top — and stay thin

Everything that makes an E2E test convincing also makes it costly, and the costs compound at the top of the pyramid:

Slowest. Real network, real disk, real rendering. A browser test that boots a page and waits on responses is thousands of times slower than a unit test. Slow tests run rarely; rarely-run tests catch bugs late.
Flakiest. An E2E test fails for every reason its dependencies can fail: a slow response, a race, a not-yet-rendered element, a shared test database left dirty by the previous run, a DNS hiccup in CI. Many of those failures have nothing to do with your code. This is not a footnote — it is the direct on-ramp to the next two pages.
Most expensive to maintain. It touches the whole system, so any change anywhere — a renamed CSS class, a new required field, a slower endpoint — can break it. The larger the surface a test covers, the more often innocent changes turn it red.

So you write few E2E tests, and you spend that scarce budget on high-value user journeys — the handful of flows where, if they broke, you would want to be paged at 3 a.m. Sign-up, log in, add-to-cart, checkout, the one report the whole business runs on. Not the forty-seven edge cases around each of them.

The rule of thumb

Cover the critical happy-path journeys end-to-end. Push edge cases down to integration and unit levels.

Ask “does the money-path work at all, through the real front door?” with a thin cap of E2E tests. Ask “what happens when the coupon is expired, the card is declined, the address has a Unicode character, the quantity is negative?” one or two levels down, where the tests are fast, deterministic, and cheap to maintain. A single E2E test proving checkout succeeds plus a dozen unit tests covering every way the price calculation can go wrong buys you far more justified confidence per second than a dozen browser tests each re-driving checkout to poke one edge case.

   RIGHT                              WRONG (inverted pyramid)
   ─────                              ────────────────────────
   1  E2E:  checkout succeeds         40 E2E: checkout with expired coupon,
   12 unit: price/tax/discount               declined card, unicode address,
            edge cases, fast                 negative qty, ... each ~5 s
   → seconds, stable                  → many minutes, flaky, ignored

The failure mode to fear is the inverted pyramid (also called the “ice-cream cone”): a suite that pushes edge cases up into slow E2E tests instead of down into fast unit tests. It runs for tens of minutes, flakes constantly, and — the fatal part — people stop trusting red and start re-running until green. A test suite that is ignored buys zero confidence, no matter how high its fidelity.

What to deliberately leave out of E2E

Choosing what not to test end-to-end is as important as choosing what to. Leave these out on purpose and cover them below:

Input-space variety. Every branch of a validation, every boundary value, every malformed input — this is exactly what test design taught you to sample, and it belongs in fast unit tests. Re-driving a full browser to try one more bad zip code is the single most common way suites become slow and flaky.
Error and failure branches. “What happens when the payment gateway times out?” is far cheaper and more reliable to test at the integration level with a controlled fake than by trying to induce a real timeout end-to-end.
Internal states and private data. If a test needs to peek at a database row or a log line to make its assertion, it has stopped asserting on user-visible behaviour and started coupling to internals — push that check down to where the internals are legitimately in scope.
Combinatorial permutations. Ten payment methods × eight countries × five discount types is 400 combinations; cover the logic once at the unit level and drive one representative journey end-to-end.

The discipline is one sentence: an E2E test answers “does this journey work at all through the real front door?” — every finer question goes to a cheaper level.

On 1 August 2012, the trading firm Knight Capital deployed new code to its servers. By most public accounts a configuration flag reused an identifier tied to old, dormant logic, and one server did not receive the update — so the fleet ran in a state no one had exercised as a whole. In roughly 45 minutes the system flooded the market with unintended orders and the firm lost on the order of $440 million, effectively ending its independence.

Whatever the exact mechanics, the testing lesson is durable: the bug lived in the fully-assembled, real-deployment behaviour of the system — the exact thing only an end-to-end test against a production-like environment can exercise. Every component may pass its own tests; the system as a whole, deployed the way it will really run, is a different subject that needs its own — thin but real — coverage.

Under the hood — why the E2E environment is where flakiness is born

An E2E test is not just slow because it does more work. It is unstable because of where it runs: a shared, stateful, networked environment, and each of those three words is a flakiness source you are about to meet in detail.

Shared — the test database, the staging cluster, the CI runner are used by many tests (and sometimes many pipelines) at once. Test A leaves a row behind; test B, expecting an empty table, fails. The bug is not in either test’s logic — it is in the shared state between them.
Stateful — the system carries state across steps: a session cookie, a cached value, a queued job. Run the same test twice and the second run starts from a different state than the first. Order-dependence and “passes alone, fails in the suite” live here.
Networked — every real call can be slow, reordered, retried, or dropped. A test that assumed a response arrived in 100 ms fails when it takes 900 ms under CI load. Nothing changed in your code; the network had a bad day.

   E2E test  ──►  [ SHARED, STATEFUL, NETWORKED environment ]
                     │          │              │
                  leftover   session /       slow / dropped /
                  rows       cache carry     reordered calls
                     │          │              │
                     └──────────┴──────────────┘
                                ▼
                    a test that fails without a code change
                                = FLAKINESS

That is not a digression — it is the map for the rest of this Part. E2E tests are where fidelity is highest and where these three environment properties bite hardest, which is exactly why the highest-fidelity tests are also the flakiest. The next two pages take this apart: Flaky Tests: Why They Break diagnoses each cause — timing, order, shared state, the network — and Curing Flaky Tests gives the concrete cures. Read this page as the reason those two exist.

Keeping the thin cap honest

A handful of E2E tests is only worth having if they stay trustworthy. Three habits keep the cap thin and green:

Own your test data. Each test should create the data it needs and clean up after itself, so it does not depend on — or corrupt — the shared environment. A test that assumes “there are already users in the database” is a flake waiting for the day someone empties it.
Assert on stable, semantic anchors, not incidental details. In a browser test, target an element by a role or an explicit data-testid, not by a brittle CSS path or the exact pixel it renders at. In an API test, assert on the contract fields you promise, not on the incidental ordering of a JSON object.
Wait on conditions, not clocks. The single largest source of E2E flakiness is sleep(2 seconds) and hope. Wait for the thing you actually need — the element to be visible, the response to arrive, the row to appear — with a bounded timeout. The cures page makes this concrete; flag it here as the first rule of writing an E2E test that stays green.

   BRITTLE                              ROBUST
   ───────                              ──────
   sleep(2); click(".btn.btn-primary")  waitFor(role=button, name="Buy").click()
   assert body[3].id == 42              assert body.order.id is present
   assumes DB already seeded            test seeds + tears down its own data

None of these buy fidelity — they buy stability at that fidelity, which is the only reason a slow, high-fidelity test is worth keeping at all.

The architect’s lens

Five questions to hold against E2E testing before you reach for it — the same altitude the pyramid argues from.

Why does it exist? Every cheaper test proves a component correct given assumptions about everything it talks to. E2E tests exist to check zero assumptions: they drive the fully-assembled system through its real door and confirm a real user’s journey actually works.
What problem does it solve? The “all green, still broken” problem in its purest form — bugs that only appear when the whole real system runs together in a real deployment: misconfiguration, UI wiring, cross-service contracts, and integration gaps no unit or single-boundary test can see.
What are the trade-offs? Maximum fidelity and persuasiveness bought at maximum cost: slowest to run, flakiest to keep green, most expensive to maintain because any change anywhere can turn them red. You trade a lot of speed and stability for a little coverage of the most important flows.
When should I avoid it? For edge cases, input validation, and error branches — push those down to fast, deterministic unit and integration tests. Reach for E2E only for a handful of critical happy-path journeys; never let it become where you test everything (the inverted pyramid).
What breaks if I remove it? You lose the only test that proves the assembled, deployed system serves a real user at all. Contracts can each hold while the whole still fails to boot, redirect, render, or wire up — a Knight-Capital-shaped gap where every part passed and the system did not.

Check your understanding

State the three load-bearing parts of the definition of an E2E test, and explain what each one rules out.
What class of bug does a browser-level E2E test catch that an API-level E2E test cannot? Give a concrete example.
E2E tests have the highest fidelity to production. Why, then, does the pyramid tell you to write few of them?
Apply the rule of thumb to a checkout flow: which single behaviour would you cover end-to-end, and which behaviours would you push down a level — and to which level?
Name the three properties of the environment E2E tests run in that make them flaky, and give one failure each produces.

Show answers

Fully assembled — nothing on the critical path is faked, or it is only a wide integration test, not E2E. Real entry point — you go in through the door a user uses (HTTP request or browser), not an internal function call. User-visible behaviour — you assert only on what comes out the door (status, body, rendered text), never on a private field or a directly-peeked row, so a refactor that keeps behaviour keeps the test green.
UI-only bugs: a button wired to the wrong handler, a form that never submits, client-side validation that blocks a valid input, a redirect loop, an element hidden by CSS. Example: the API returns 200 and writes the order row correctly, but the “Buy” button’s click handler is broken, so no real user can ever place the order — an API-level test passes, a browser test fails.
Because fidelity is bought with cost. E2E tests are the slowest (real network/disk/rendering), the flakiest (they fail for every reason their real dependencies can fail), and the most expensive to maintain (any change anywhere can break them). For a fixed time budget you can run thousands of unit tests or a few dozen E2E tests — so you spend the scarce E2E budget only on the highest-value journeys.
Cover “checkout succeeds through the real front door” with one (or a few) E2E tests — the money path. Push edge cases down: expired coupon, declined card, negative quantity, Unicode address, tax/discount math → unit tests (fast, deterministic) for the pure logic, and integration tests for the ones that cross a real boundary (e.g. the payment client against a real/stubbed gateway). Fast and cheap where the variety lives; thin and real where “does it work at all” lives.
Shared (a leftover row from one test fails another that expected an empty table), stateful (a session/cache carried across steps makes a second run start differently and fail), networked (a call that took 900 ms instead of 100 ms under CI load trips a too-tight assumption). None involves a code change — all three are the on-ramp to the flakiness pages that follow.