Regression, Smoke & Sanity — Guarding What Already Works

The previous page, Functional Testing, asked one question: does the feature do the right thing? That question is about behavior you are building now. This page is about behavior you built before — and would very much like to keep.

Most software damage does not come from the feature you are writing. It comes from the ten features you are not thinking about while you write it — the ones your change quietly disturbs. Regression, smoke, and sanity testing are three fast checks that guard already-working behavior. They sound similar and are constantly confused, but each answers a genuinely different question, at a different breadth, run at a different moment. Getting them straight is one of the highest-leverage things a tester can learn.

Three questions, not three synonyms

Before any definitions, hold the three questions in your head. They are what actually distinguish these checks:

  Regression : "Did my change break something that used to work?"
  Smoke      : "Is this build fundamentally alive — worth testing at all?"
  Sanity     : "Is this ONE area I just touched behaving correctly?"

Everything else — breadth, depth, when you run it, how automated it is — falls out of which question you are asking.

Regression testing — proving you broke nothing

Regression testing is re-running existing checks to prove that a change did not break previously-working behavior. A regression is exactly that: a thing that used to work and now doesn’t. The test itself is nothing new — it is your old tests, run again, after a change.

The mechanism is almost embarrassingly simple, and that is the point. Every time you fix a bug or add a feature, you leave behind a test that pins the correct behavior. Over months, those tests accumulate into a regression suite: a large, boring, comprehensive record of “things that are supposed to be true.” Change the code, run the suite, and any test that flips from green to red is a regression — behavior you broke without meaning to.

   feature A test  ─┐
   feature B test  ─┤
   bugfix #212 test ┼──►  run ALL of them after every change
   bugfix #240 test ─┤        │
   feature C test  ─┘        red? ──► you regressed something. stop.

This is where the payoff of automation becomes overwhelming. A regression suite is the same checks, run over and over, forever. A human doing that by hand is slow, expensive, and — worse — gets bored and stops looking carefully. A machine runs 3,000 assertions in ninety seconds and never gets bored. The entire economic case for automated testing is regression: you pay to write the check once, and it defends that behavior on every future change at nearly zero marginal cost. The kvlite integration tests in this repo (rust/kvlite/tests/server.rs) are a regression suite in miniature — every time the storage engine changes, tcp_set_get_del_roundtrip re-proves that a client can still SET, GET, and DEL over the wire.

By the numbers — the cost of not having a suite

Regression suite:   3,000 assertions, runs in  90 seconds, cost per run ≈ $0
Manual re-check:     40 core flows, ~4 minutes each, by a person
                     = ~2.7 hours per release, done maybe once, tiredly

Ship 20 times a month:
  automated  = 20 × 90s   =  30 min of machine time, every path checked
  manual     = 20 × 2.7h  =  54 hours of human time — so it doesn't happen

The suite doesn’t just save time. It changes what’s possible: shipping often only stays safe if the regression check is cheap enough to run every time.

It actually happened — Ariane 5, June 1996

On its maiden flight (4 June 1996), the Ariane 5 rocket veered off course and self-destructed about 37 seconds after launch, destroying roughly US$370 million of payload. The root cause was reused code that nobody re-tested in its new context. A guidance routine carried over from the older, slower Ariane 4 converted a 64-bit horizontal-velocity value into a 16-bit integer. Ariane 5 flew faster, the number overflowed the 16-bit slot, the software raised an unhandled exception, and the flight computer shut down.

The code was not buggy for Ariane 4. It became a bug the moment its assumptions changed and no regression check re-validated it against the new flight profile. That is the entire regression lesson: reused, “already-working” code is only already-working under the conditions you last tested it. Change the surrounding world and you must re-run the checks — or you are trusting an assumption you never re-examined.

Smoke testing — is the build even alive?

Smoke testing is a shallow, broad check that a build is fundamentally functional before anyone spends time testing it deeply. The name comes from hardware: power on a new board and if smoke pours out, you stop — no point probing individual pins. In software, a smoke test asks the crudest possible version of “does this thing work at all?”

• Does the app start?
• Does the home page load (HTTP 200, not 500)?
• Can a user log in?
• Does /healthz come back OK and can you reach the database?

That’s it. Smoke tests are broad (they touch many major areas) but deliberately shallow (they check that each area is present and responding, not that it is correct in detail). A smoke test does not verify that the checkout total is computed correctly; it verifies that the checkout page renders at all.

The value is as a gate. A smoke suite runs in seconds to a couple of minutes, and its job is to fail fast on a build that is dead on arrival. If login is completely broken, there is no reason to run a 90-minute regression suite that will fail on every login-dependent test and drown you in noise. Smoke first; if it passes, the build has earned the deeper, slower testing.

   new build ──► SMOKE (broad, shallow, ~60s)
                    │
              pass? ├── yes ──► worth running full regression / deeper tests
                    └── no  ──► reject the build now; don't waste an hour

This is why smoke tests are the classic CI gate: the first stage of a pipeline, run on every commit, cheap enough to block a merge that broke the world.

A good rule of thumb for what belongs in a smoke suite:

  belongs in smoke:            does NOT belong in smoke:
  ─────────────────            ────────────────────────
  app boots                    "the discount is 12.5% not 12%"
  home page returns 200        every edge case of the parser
  login succeeds               all 40 error messages render
  /healthz reaches the DB      pixel-perfect layout on 6 browsers
  one create/read round-trip   the 3,000-assertion regression pass

  → a handful of the most load-bearing paths, each checked once, shallow

If your smoke suite grows past a couple of minutes, it has stopped being a smoke suite and started becoming a slow gate — split the deep checks back out into regression.

Sanity testing — is this one area sane?

Sanity testing is a narrow, deep check that one specific changed area behaves correctly — skipping the full suite. It is the mirror image of smoke. Where smoke is broad-and-shallow across the whole build, sanity is narrow-and-deep on one region.

The situation: you just shipped a hotfix to the tax-calculation code. You do not need to re-verify login, search, or the profile page — nothing there changed. You need to know, in detail, that tax calculation is now sane: the fix works, and it didn’t obviously wreck its immediate neighbors. So you run a focused set of deep checks on that one area and skip everything else.

Sanity testing is a judgment call, often quick and semi-formal, made to answer “is it reasonable to proceed with this specific change?” — not “is the whole system correct?”. It is what you run on a hotfix when a full regression pass is too slow and mostly irrelevant to what you touched.

   SMOKE            SANITY           full REGRESSION
   ───────          ───────          ───────────────
   ██ ██ ██ ██      ░░ ░░ ██ ░░      ██ ██ ██ ██
   ░░ ░░ ░░ ░░      ░░ ░░ ██ ░░      ██ ██ ██ ██
   ░░ ░░ ░░ ░░      ░░ ░░ ██ ░░      ██ ██ ██ ██
   broad, shallow   narrow, deep     broad, deep
   (touch all,      (one area,       (everything,
    surface only)    thoroughly)      thoroughly)

The three side by side

The clean way to keep these straight is a table on three axes — breadth, depth, and when run:

Check	Breadth	Depth	The question	When it’s the right tool
Smoke	Broad (all major areas)	Shallow (present & responding)	Is the build alive at all?	CI gate — every commit, before deeper testing
Sanity	Narrow (one changed area)	Deep (that area, thoroughly)	Is this one change sane?	Hotfix — verify the patch, skip the rest
Regression	Broad (whole system)	Deep (every pinned behavior)	Did I break anything old?	Before release — the full, slow, comprehensive pass

Read the table as a progression of trust. Smoke lets a build in the door. Sanity clears one specific change fast. Full regression is the thorough final exam you run before you ship to everyone. They are not competitors — a mature pipeline uses all three, at different moments, for different budgets of time.

A useful mental model is a time budget. Each check spends a different amount of time to buy a different amount of confidence:

   confidence
      ▲
 full │                                    ● full regression (90 min)
      │                                   ╱   "nothing old is broken"
      │                      ● sanity   ╱
      │                     ╱ (5 min)  ╱
      │       ● smoke      ╱  "this    ╱
      │      ╱ (60 s)     ╱   change   ╱
      │     ╱ "build is  ╱    is sane" ╱
      │    ╱  alive"    ╱             ╱
      └───────────────────────────────────────►  time spent

You reach for the cheapest check that answers the question in front of you — and only pay for the expensive one when the stakes (a release to all users) justify it.

A note on naming — “smoke” vs “sanity” in the wild

Be warned: teams use these two words loosely and sometimes swap them. Some shops call any quick pre-flight check a “smoke test,” including narrow sanity-style ones; others use “sanity check” as a synonym for smoke. The words matter less than the two ideas underneath them, which are genuinely distinct: broad-and-shallow across a whole build versus narrow-and-deep on one changed area. When you join a team, confirm which idea they attach to each word rather than assuming — the concepts are stable even where the vocabulary drifts.

Under the hood — how a suite gets partitioned

In practice these three checks are frequently the same test files, tagged and selected differently rather than written separately. A test runner lets you label tests and run a subset:

# pytest-style tags select which slice runs when.
@pytest.mark.smoke        # cheap, broad — runs on every commit
def test_home_page_loads(client):
    assert client.get("/").status_code == 200

@pytest.mark.tax          # the "tax" area — the sanity slice for a tax hotfix
def test_tax_rounds_half_up(calc):
    assert calc.tax(cents=1_001, rate=0.10) == 101  # not 100

# (no tag) — part of the full regression suite, run before release
def test_checkout_applies_coupon_before_tax(cart):
    ...

  CI on every commit :  pytest -m smoke          # broad, shallow gate
  Hotfix to tax      :  pytest -m tax            # narrow, deep sanity
  Before a release   :  pytest                   # everything = regression

Same assertions, different selection. “Smoke,” “sanity,” and “regression” are often less about what a test contains and more about which slice you choose to run, and when.

The one thing that kills a regression suite: flakiness

A regression suite only works if a red result means something. The failure mode that quietly destroys suites is flaky tests — tests that pass and fail on the same code, usually because of timing, ordering, shared state, or a network hiccup. The kvlite server test avoids the classic trap by binding to port 0 and letting the OS hand it a free port, so two test runs never collide on a fixed port. Ignore that discipline and you get intermittent failures.

The danger of flakiness is not the wasted re-run. It is what it does to humans: once “the suite is red” sometimes means “nothing is actually wrong,” people start re-running until it’s green and stop reading failures. A regression suite that everyone has learned to ignore protects nothing. Treat a flaky test as a broken test — quarantine or fix it — because a suite you don’t trust is worse than no suite, since it costs time and gives false comfort.

Where these three sit relative to each other

If you are choosing which check a given change needs, the decision is almost mechanical — and it’s the whole subject of a later page, Choosing What a Change Needs:

   what did you change?
        │
        ├─ a whole new build, about to be tested?  ──► SMOKE first
        │
        ├─ one isolated area, urgent hotfix?        ──► SANITY on that area
        │
        └─ anything, about to ship to all users?    ──► full REGRESSION

The architect’s lens

• Why does it exist? Because a change never lives in isolation — it perturbs code you aren’t looking at. These three checks exist to defend already-working behavior cheaply, so that “I added a feature” doesn’t silently mean “I broke two old ones.”
• What problem does it solve? Wasted time and undetected breakage. Smoke stops you deep-testing a dead build; sanity stops you re-running an hour of irrelevant tests for a one-line hotfix; regression stops old behavior from rotting as the system grows.
• What are the trade-offs? Coverage costs time and maintenance. A full regression suite is slow and must be kept green, or it becomes noise everyone ignores. Smoke and sanity trade thoroughness for speed — they can pass while a subtle bug survives in an area they didn’t probe.
• When should I avoid it? Never skip regression before a real release. But do skip the full suite for a tiny, isolated hotfix where sanity is enough — and don’t gold-plate a smoke suite into a slow one, or it stops being a fast gate.
• What breaks if I remove it? Remove regression and you get Ariane 5: reused, “trusted” code that nobody re-validated after its world changed. Remove smoke and CI wastes an hour testing builds that were dead on arrival. Remove sanity and every trivial hotfix drags the whole suite behind it.

Check your understanding

1. In one sentence each, state the distinct question that regression, smoke, and sanity testing answer.
2. On the axes of breadth and depth, place all three checks. Which one is broad-and-shallow, which is narrow-and-deep, and which is broad-and-deep?
3. Why is automation the decisive factor for regression testing specifically, more than for a one-off exploratory session?
4. A production hotfix must go out in 20 minutes and the full regression suite takes 90. Which check do you run, and why is skipping the rest defensible here?
5. Explain how the Ariane 5 failure is fundamentally a regression lesson, even though the reused code had never been “wrong” before.

Show answers

1. Regression: did my change break something that previously worked? Smoke: is this build fundamentally functional — alive enough to be worth testing? Sanity: is this one specific area I changed behaving correctly?
2. Smoke is broad-and-shallow (touches all major areas, only checks they respond). Sanity is narrow-and-deep (one changed area, checked thoroughly). Regression is broad-and-deep (the whole system, every pinned behavior, thoroughly).
3. A regression suite is the same checks re-run on every future change, forever. That repetition is exactly what machines do cheaply and reliably and humans do slowly, expensively, and carelessly. Automation makes the marginal cost of re-checking near zero, which is what makes frequent, safe shipping possible.
4. Run a sanity check: narrow and deep on the changed area. Skipping the rest is defensible because nothing else was touched, so re-running unrelated tests mostly wastes time you don’t have — sanity buys reasonable confidence in the actual change fast. (You’d still run the full regression suite afterward, off the critical path.)
5. The Ariane 4 code was correct under Ariane 4’s conditions. Ariane 5 flew faster, its velocity value overflowed a 16-bit conversion, and no test re-validated the reused code against the new flight profile. That is the regression lesson exactly: “already-working” code is only proven under the conditions you last tested — change the surrounding world and you must re-run the checks, or you’re trusting an assumption you never re-examined.