End-to-End Tests and a Coverage Goal

On the previous page we covered Snip’s pure core in microseconds and drove its endpoints against a real database — including the concurrency race. That gave us two of the three kinds of confidence a feature needs. What we still lack is proof that the whole thing works from the outside, the way a user actually hits it: a deployed process, over the network, exercising every layer at once. And we have no number telling us how much of the code our tests even touch.

This page adds both. First, a thin layer of end-to-end (e2e) tests that talk to the running service over HTTP and verify the full shorten-then-redirect round trip. Then a coverage goal — a defensible target and, just as important, an argument for why 100% is the wrong number to aim at.

What “end-to-end” actually means here

An integration test starts the server inside the test process and talks to it in memory or over a loopback socket. Useful — but it still runs the code in a test harness, with test wiring, often against a test-only database on the same machine.

An end-to-end test removes that last bit of pretend. It runs against the service exactly as deployed: the real process, listening on its real port, behind whatever proxy or router sits in front of it, backed by the same kind of Postgres and Redis it will use in staging. The test is a plain HTTP client. It knows nothing about Snip’s internals — only its public contract, the two endpoints a real caller sees.

   ┌────────────────────────────────────────────────────────┐
   │  e2e test process                                        │
   │     an HTTP client, no knowledge of internals            │
   └───────────────┬────────────────────────────────────────┘
                   │  real HTTP, over the network
                   ▼
   ┌────────────────────────────────────────────────────────┐
   │  DEPLOYED Snip service (the real binary/container)       │
   │     router → app → Postgres + Redis                      │
   └────────────────────────────────────────────────────────┘

The value of that last layer is that it catches the failures nobody’s unit or integration test can see, because they live between the pieces: a reverse proxy that strips the Location header, a container that binds the wrong port, a config file that points at the wrong database, TLS that is misconfigured, a health check the orchestrator reads differently than you expected. Each of those passes every in-process test and still takes the service down. E2e is the only layer that exercises the deployment itself.

One or two e2e tests — no more

We write very few of these. Here is the whole shorten-then-redirect round trip against a deployed base URL:

# tests/e2e/test_roundtrip.py — runs against a DEPLOYED Snip, over HTTP
import os, requests

BASE = os.environ["SNIP_BASE_URL"]   # e.g. https://snip.staging.example.com

def test_shorten_then_redirect_end_to_end():
    # ACT: create a short code through the real, deployed API
    long_url = "https://example.com/e2e/probe"
    r = requests.post(f"{BASE}/shorten", json={"url": long_url}, timeout=5)
    assert r.status_code == 201
    code = r.json()["code"]

    # ACT: follow the short code, exactly as a browser would — but don't auto-follow,
    # so we can inspect the redirect itself
    r2 = requests.get(f"{BASE}/{code}", allow_redirects=False, timeout=5)

    # ASSERT the user-visible contract: a redirect back to the original URL
    assert r2.status_code == 302
    assert r2.headers["Location"] == long_url

A second one is worth having: a smoke check that the service is even alive and wired to its dependencies, run first so a total outage fails fast with a clear message instead of a confusing redirect assertion.

def test_health_and_metrics_are_up():
    assert requests.get(f"{BASE}/healthz", timeout=5).status_code == 200
    assert requests.get(f"{BASE}/metrics", timeout=5).status_code == 200

That is close to all the e2e we want. Two round trips and a smoke check. The instinct to add “just one more” e2e for every case is exactly the instinct to resist — and the next section says why.

The test pyramid

The shape of a healthy suite has a name: the test pyramid. Many fast unit tests at the base, fewer integration tests in the middle, a very thin cap of e2e tests on top.

              ▲   slow, brittle, high-fidelity
             ╱ ╲          e2e  (a few)
            ╱   ╲     ────────────────────
           ╱     ╲    integration (some)
          ╱       ╲  ──────────────────────
         ╱  unit   ╲  (many, fast, isolated)
        ╱___________╲
              ▼   fast, stable, low-fidelity

The pyramid is not an aesthetic preference; it falls out of a cost/fidelity trade-off. As you climb, each test buys you more realism — an e2e test proves the deployed system works, which no unit test can — but each test also costs more to run, breaks more often for reasons unrelated to your change (a slow network, a cold container, a flaky DNS lookup), and points less precisely at the fault when it fails. A red unit test names the broken function. A red e2e test tells you only that something in the whole stack is wrong, and now you go hunting.

So you push every check as far down the pyramid as it will go. Test URL validation in a unit test, not an e2e test. Test the storage contract in an integration test, not an e2e test. Reserve e2e for the one thing only it can verify: that the real, deployed, fully-wired system serves a real request.

Why inverting the pyramid hurts

The failure mode has a name too: the ice-cream cone — a fat layer of slow e2e tests, a little integration, almost no unit tests.

        ╲   e2e (many)  slow, flaky   ╱   ← the "ice-cream cone":
         ╲_________________________╱        an inverted pyramid
          ╲   integration (some) ╱
           ╲___________________╱
            ╲  unit (few)     ╱
             ╲_______________╱

It happens naturally: e2e tests are easy to think of (“just click through it”) and feel the most convincing, so a team keeps adding them until the whole suite is e2e. The result is a suite that:

Runs in minutes, not seconds, so nobody runs it before pushing — the fast feedback loop the whole book is about is gone.
Fails randomly. With enough network-dependent tests, some flake on every run for reasons unrelated to any code change. People learn to re-run red builds until they go green — and the day a real bug turns the build red, they re-run that one too, and ship it.
Localizes nothing. A hundred e2e failures all say “the request didn’t return what I expected,” and you debug each by hand.

The pyramid is the shape that keeps the suite fast, stable, and diagnostic. Inverting it trades all three away for a false sense of realism.

Coverage: what it measures

Now the number. Code coverage answers one narrow question: which lines and branches of the code were executed while the tests ran? You instrument the code, run the suite, and get a report.

Two flavors matter, and the difference is the whole point:

Line coverage — what fraction of executable lines ran at least once.
Branch coverage — what fraction of decision outcomes ran: for every if, did tests exercise both the true and the false path? For every match/switch, was each arm hit?

Branch coverage is the stricter, more honest number, because a line can be “covered” while a branch through it never runs. Consider:

def redirect(code, store):
    row = store.get(code)
    if row and not row.expired:      # two conditions, several outcomes
        return Redirect(row.url)
    return NotFound()                # is THIS line ever reached in a test?

A single test that shortens then redirects a fresh code executes every line here — 100% line coverage — while never once taking the NotFound path or the row.expired path. Line coverage says “done.” Branch coverage says “you tested one of at least three outcomes.” The untested branches are exactly the risky ones: the not-found case, the expired case. Read the branch report, not the line report, to find the holes that matter.

Under the hood — reading a coverage report

A coverage report is per-file, with a summary. Learn to skim it for the low-branch, high-risk rows rather than the top-line percentage:

File                 Lines   Miss  Line%   Branch%   Missing branches
------------------------------------------------------------------
core/normalize.py      42      0   100%     100%     —
core/generate.py       18      0   100%     100%     —
http/shorten.py        60      3    95%      88%     shorten.py:41 (dup-code retry)
http/redirect.py       34      6    82%      60%     redirect.py:12 (expired), :19 (404)
store/postgres.py      90     14    84%      71%     postgres.py:55 (unique-violation)
------------------------------------------------------------------
TOTAL                 244     23    91%      78%

The 91% line number is almost noise. The story is in the missing branches column: the expired-URL path, the 404 path, and the unique-violation retry are untested — and those are precisely the paths from our risk map that break under bad input and concurrency. A coverage report is most useful read this way: as a diagnostic that points at unexercised risky branches, not as a grade.

Setting a defensible goal

So what number do you put in CI as the gate? A good policy has two parts:

Overall line/branch coverage ≥ 80%, and the critical paths at 100%.

The 80% is a floor for the codebase as a whole — high enough to catch large swaths of dead-untested code, low enough to leave room for the parts that genuinely aren’t worth testing (a debug-print branch, a defensive else that “can’t happen”). The critical paths at 100% is the part that actually protects users: the shorten-then-redirect round trip, the URL-validation rules, and the uniqueness invariant under concurrency — those you require to be fully covered, because a bug there is a bug every user hits.

Coverage climbing from 80% → 100% on Snip (illustrative):

  80% → 90%   covers the remaining ordinary branches
              effort: ~1 day     bugs likely found: some
  90% → 95%   covers rare error paths, odd inputs
              effort: ~2 days     bugs likely found: few
  95% → 100%  covers "can't happen" defensive code,
              unreachable-in-practice branches, platform quirks
              effort: ~1 week     bugs likely found: ~none

The curve is not linear. The first 80% is cheap and finds most bugs.
The last few percent cost the most and find the least — and often
get "covered" by tests with no real assertions, just to move the number.

Why 100% is the wrong target

Chasing 100% coverage sounds rigorous and is quietly harmful:

It optimizes the wrong thing. Once the number is the goal, people write tests that execute a line without checking anything, purely to color it green. The metric goes up; confidence does not.
It has terrible marginal returns. As the table shows, the last few percent are the “can’t happen” branches — you spend the most effort testing the code least likely to break, while a whole class of bugs (concurrency, integration, deployment) lives outside what line coverage even measures.
It punishes honest defensive code. A default: arm that raises “unreachable state” is good engineering, and it is nearly impossible to cover without contorting the code. A 100% mandate pressures people to delete safety nets to satisfy a counter.

80% with critical paths at 100% keeps the incentive honest: cover the code that matters completely, cover the rest broadly, and don’t spend a week gaming the last 3%.

Coverage is a floor, not a proof

Here is the most important idea on the page, and the one most often forgotten: coverage measures what ran, not what you checked. It cannot see your assertions. A test can execute every line and branch and assert nothing meaningful — and it will report 100% coverage while catching zero bugs.

# 100% coverage of redirect(), and completely worthless:
def test_redirect_runs():
    resp = redirect("abc", store)
    assert resp is not None    # ← executes the code, verifies nothing real

That test touches every line redirect runs. Coverage tools count it as full coverage. But assert resp is not None would pass whether the function returned the right URL, the wrong URL, or a 500 error page — as long as it returned something. The line is green. The behavior is unverified. High coverage with weak assertions still misses bugs, and it misses them while wearing a badge that says everything is tested.

So hold coverage in its proper place:

It is a floor: coverage near zero is a red flag — code that no test has ever executed is code no test can be protecting. That signal is real and worth gating on.
It is a diagnostic: the missing-branch report points you at untested risky paths, as we saw above. That is its best use.
It is not a proof of correctness. The proof comes from assertions on behavior — the arrange-act-assert checks from the last page that verify the right output, not merely an output. Coverage tells you the code ran; only a strong assertion tells you it ran correctly.

A useful mental model: coverage answers “did we look here?” — never “is what we found correct?” You still need good tests behind the number.

In several widely reported outages over the past decade, code that was covered by an automated suite still failed in production, because the tests exercised the code path without asserting the property that actually mattered. A frequently cited pattern: input-handling code with high line coverage whose tests confirmed the function “returned a response” but never asserted the response was correct for a malformed or edge-case input — so a boundary bug sat behind a green suite for months. Exact figures vary by account, but the lesson is stable: a covered line is not a verified line. If the assertion does not encode the behavior a user depends on, the coverage number is decoration. Ask of every test not “does it run the code?” but “would it go red if the behavior were wrong?”

The architect’s lens

Why does it exist? E2e tests and coverage exist to close two blind spots the lower layers leave: whether the deployed, fully-wired system serves a real request, and how much of the code the suite touches at all.
What problem does it solve? E2e catches between-the-pieces failures (proxy, port, config, TLS) that pass every in-process test; coverage flags untested risky branches so you find the holes before production does.
What are the trade-offs? E2e is slow, flaky, and points poorly at the fault, so you keep it to a thin cap; coverage is cheap to gather but easy to misread as a correctness score when it only measures execution.
When should I avoid it? Avoid pushing a check into e2e that a unit or integration test could catch faster and more precisely, and avoid gating on 100% coverage — it rewards assertionless tests and punishes honest defensive code.
What breaks if I remove it? Drop e2e and deployment/wiring bugs ship undetected; drop coverage and whole files can sit entirely untested with nothing warning you they were never exercised.

Check your understanding

What does an end-to-end test verify that an in-process integration test cannot, and why does that justify running the service exactly as deployed?
Explain the test pyramid and the ice-cream cone. What three concrete problems does inverting the pyramid cause?
A file shows 100% line coverage but 60% branch coverage. What does that gap tell you, and which report should you read to find the risky untested paths?
State a defensible coverage policy for Snip and give two reasons why aiming for 100% overall is the wrong target.
A test executes every line of redirect() and asserts only resp is not None. What is coverage reporting, what is it not proving, and what would make the test actually protective?

Show answers

An e2e test verifies the real, deployed, fully-wired system — the actual process on its real port, behind its real proxy, against real Postgres/Redis — serves a real HTTP request correctly. Integration tests still run the server inside the test process with test wiring, so they cannot catch between-the-pieces failures like a proxy stripping the Location header, a wrong bound port, or a misconfigured database URL. Only running it as deployed exercises the deployment itself.
The pyramid is many fast unit tests at the base, fewer integration tests, a very thin cap of e2e — the shape that keeps the suite fast, stable, and diagnostic, because you push each check as far down as it will go. The ice-cream cone inverts it: mostly slow e2e tests. Inverting causes: (a) the suite runs in minutes so nobody runs it before pushing; (b) it flakes randomly so people re-run red builds until green — and ship real bugs that way; (c) it localizes nothing — every failure just says “the request didn’t return what I expected.”
The gap means every line ran, but for the if/match decisions only some outcomes were exercised — e.g. the true path was hit but the false/error path never was. Those unexercised branches (expired, 404, retry) are usually the risky ones. Read the branch coverage report and its missing-branches column, not the line percentage, to find them.
Policy: overall line/branch ≥ 80%, critical paths (shorten→redirect, URL validation, uniqueness under concurrency) at 100%. 100% overall is wrong because (a) it has terrible marginal returns — the last few percent are “can’t happen” defensive branches costing the most effort to find the fewest bugs; and (b) it incentivizes assertionless tests that execute lines to color them green without verifying behavior, and pressures deleting good defensive code to satisfy the counter.
Coverage is reporting that every line of redirect() executed — full execution coverage. It is not proving the function returned the right URL — resp is not None passes for the correct URL, the wrong URL, or an error page alike. To make it protective, assert on the behavior a user depends on: the status is 302 and Location equals the original URL — an assertion that would go red if the behavior were wrong.