Choosing What a Change Needs — A Decision Guide

Seven pages handed you seven lenses. Functional for wrong answers, regression, smoke and sanity for broken-what-worked, performance for too-slow, security for exploitable, accessibility, usability and compatibility for unusable, and exploratory for the paths no script imagined. Each named the risk it buys down. What we have not done is the thing you do a hundred times a week: look at one actual change and decide, deliberately, which of those lenses it warrants.

This page is that decision. It is the payoff the whole part was built toward — not “here are the types” but “here is how to choose.” The method is small enough to run in your head before you write a single assertion, and it beats both of the lazy defaults: run everything (slow, and you still miss the risk that isn’t in a suite) and test what’s easy (fast, and blind to exactly the risks that hurt).

The one idea: testing is a finite budget spent to buy down risk

Every test costs something real — time to write, time to run on every future change, time to keep green when it breaks for boring reasons. That budget is finite. So testing is not a virtue you apply maximally; it is spending, and the only question is whether each test buys down more risk than it costs.

That reframes both failure modes as the same mistake — misallocation:

• Over-testing a low-risk change is waste. A full load test on a help-text typo spends budget that bought nothing, and worse, it trains the team to route around slow, pointless gates.
• Under-testing a high-risk change is a bet you didn’t know you made. Shipping a password-hashing change with only the unit tests that happened to exist is cheap today and expensive the day it fails.

   risk of the change ─►  ◄─ cost of the tests

   low risk,  heavy testing   =  waste        (spent budget, bought nothing)
   high risk, light testing   =  exposure     (kept budget, kept the risk)
   risk matched to testing     =  the goal     (spend where risk lives)

The skill is matching — reading how much risk a change carries and of which kinds, then spending exactly there. Not evenly. Not maximally. Deliberately.

The procedure

Four steps, run in order, for any change:

  1. CLASSIFY the change   — what kind of change is this?
  2. MAP to risk classes   — what could this kind of change break?
  3. SCORE each risk       — likelihood × cost, on THIS change
  4. PICK type × level      — the cheapest tests that cover the top risks

Step 1 — Classify the change

Most changes fall into one of a handful of shapes, and the shape is a strong prior for the risks it carries. You are pattern-matching, not proving:

Change kind	What it is	Its characteristic risk
Bug fix	Correcting wrong behavior	Regression (fix breaks a neighbor); did the fix actually fix it?
Hotfix	An urgent bug fix, on the clock	Same as bug fix, but no time for the full suite — sanity + targeted regression
New feature	Behavior that didn’t exist	Functional (does it do the right thing?), plus whatever the feature touches — security, performance, accessibility
Refactor	Change structure, not behavior	Regression is the whole story — behavior must be byte-for-byte unchanged
Config change	A value, flag, or env setting	Deploy-time surprises: does the app still boot? Is the new value valid? Often no code path is tested at all
Dependency bump	New version of a library or base image	Regression (the Ariane 5 risk: reused code, new world) + security (new CVEs, new or fixed)

The classification is not bureaucracy — it is the fastest way to recall which risks this kind of change has historically carried, so you don’t have to re-derive them from scratch each time.

Step 2 — Map to risk classes

Now attach the specific risks. A refactor’s risk is almost entirely regression; a new public endpoint fans out across several classes. The map is not one-to-one — one change can light up four lenses:

  change kind        risks it typically introduces
  ───────────        ─────────────────────────────
  refactor       ──► regression (behavior must not move)
  config bump    ──► "does it boot?", value-validity, sometimes security (a flag opens a door)
  dep bump       ──► regression + security (new CVEs) + occasionally performance
  bug fix        ──► "does it fix it?" (functional) + regression (neighbors)
  new endpoint   ──► functional + security (new input/trust boundary) + performance (new load)
  UI change      ──► accessibility + compatibility + usability + regression

Step 3 — Score each risk

For each risk on the map, score it likelihood × cost for this change specifically — the same gut-check the overview introduced. A new field on an internal admin form and the same field on a public checkout carry different security cost even though the code is nearly identical. Scoring is what stops the map from becoming a checklist: not every risk a change could have is worth spending on.

Step 4 — Pick type and level together

This is the connection back to the whole book, and the step people skip. Choosing a type is only half the decision; you also choose the level — unit, integration, or end-to-end — from the Levels part. They are orthogonal axes, and you pick a point on both:

                 TYPE (what you look for)
                 functional  regression  security  performance  a11y …
        unit        ▓            ▓            ·          ▓          ·
 LEVEL  integ.      ▓            ▓            ▓          ·          ·
        e2e         ▓            ▓            ▓          ▓          ▓

The rule of thumb: buy down each risk at the cheapest level that can actually see it. A rounding bug in a tax function is a functional risk best caught by a unit test — fast, precise, no server needed. An auth-bypass is a security risk that a unit test structurally cannot see; it needs at least integration level, where the real middleware runs. Picking the type without the level gives you a security test that runs at the wrong scope and proves nothing.

Worked examples

The procedure is only real when you watch it run. Three changes, three very different selections.

A hotfix: sanity + targeted regression

Production is returning a 500 on one endpoint; the fix is a two-line null check. You have twenty minutes.

  classify : hotfix (urgent bug fix)
  risks    : does it fix the 500? (functional)  +  did it break a neighbor? (regression)
  score    : fix-works = high value, must confirm.  broad regression = low likelihood
             (two lines, one function), and no time.
  pick     : functional unit test on the null case  (fast, proves the fix)
           + SANITY: deep checks on that one endpoint  (its immediate neighbors)
           + SMOKE after deploy: does the app still boot and serve?
           SKIP the 90-minute full regression on the hot path; run it after, off-path.

Note what you did not do: no load test, no security review, no accessibility audit. The change touched none of those risk surfaces. Spending there would be pure waste on the clock.

A new public endpoint: functional + security + load

You’re adding POST /v1/exports — authenticated, takes a JSON body, kicks off a report, returns a job id. It’s public and internet-facing.

  classify : new feature (new public surface)
  risks    : functional (right job created? right id back? right errors on bad input?)
           + security (new input + new trust boundary: authz, injection, rate limits)
           + performance (new load path; a heavy report could exhaust workers)
  score    : all three high — public, authenticated, does real work.
  pick     : FUNCTIONAL  — integration tests: valid body → job, malformed body → 400,
                           wrong auth → 401/403.  Like kvlite's `tcp_set_get_del_roundtrip`,
                           drive the real endpoint over the wire and assert the reply.
           + SECURITY    — integration/e2e: authz on someone else's export, injection in
                           the body, does rate-limiting hold?
           + PERFORMANCE — load test the endpoint under expected concurrency; a spike test
                           if launches can stampede.
           + REGRESSION  — run the suite; a new route can shadow an existing one.

One change, four lenses — and each at the level where its risk is visible: functional and security at integration/e2e (the real auth stack must run), performance at the system level (one request proves nothing about load).

A UI change: accessibility + compatibility + usability

You’re redesigning the checkout button — new color, new copy, moved up the page. No backend change at all.

  classify : UI change (feature, presentation only)
  risks    : accessibility (contrast of the new color? still keyboard-reachable? labeled?)
           + compatibility (renders on the browsers/devices real customers use?)
           + usability (does moving it help or confuse the known flow?)
           + regression (the click still triggers the same checkout call)
  score    : a11y high (checkout is load-bearing + legal exposure); compat high (revenue
             path across a device zoo); usability medium; functional LOW (no logic changed).
  pick     : ACCESSIBILITY — automated contrast + keyboard/screen-reader checks
           + COMPATIBILITY — render across the supported browser/device matrix
           + USABILITY     — a quick task-based observation if the move is significant
           + REGRESSION    — confirm the click still fires checkout (a thin e2e)
           barely any FUNCTIONAL testing — the logic didn't move; the *presentation* did.

Three changes, and no two share a selection — because their risk profiles don’t. That is the whole method: same tool-belt, different picks, driven by the change in front of you.

By the numbers — a change’s test budget, allocated

Treat a review as ~4 hours of test-writing budget and allocate it against scored risk, rather than spreading it evenly:

  change = "new public POST /v1/exports"
  risk (likelihood × cost)        budget → tests
  ────────────────────────        ──────────────
  security     HIGH × HIGH  = 9   ~40%  authz, injection, rate-limit (integration/e2e)
  functional   HIGH × MED   = 6   ~30%  happy + error paths (integration)
  performance  MED  × HIGH  = 6   ~25%  load + spike (system)
  regression   LOW  × MED   = 2   ~5%   run existing suite (free-ish)
  accessibility  —  (no UI)       0%    not on this change's surface

  4h spent where the risk*cost is, not 48 min evenly across five types.

The figures are illustrative — the habit is the point: score, then allocate, rather than test what’s nearest to hand.

It actually happened — Knight Capital, August 2012

On 1 August 2012, the trading firm Knight Capital deployed new order-routing code to its production servers. A deploy step reportedly missed one of eight servers, and a repurposed feature flag switched on dormant, years-old code on the machines that did update. In about 45 minutes the mismatched fleet sent millions of unintended orders into the market; the firm took a loss commonly cited around US$440 million and was effectively finished within days.

Read through this page’s lens, it was a misclassified change. A deploy with a flag flip is not “just config” — it is a config and dependency-activation change, whose characteristic risks are “does every node actually get the new state?” and “what dormant path does this flag wake up?” Those risks warranted a deploy-verification and a regression check on the reactivated code — exactly the tests a “it’s only a config flag” framing skips. The lesson is not “test more.” It is classify the change correctly, because the classification is what tells you which risks to buy down.

Why deliberate choice beats the two defaults

Both lazy defaults feel responsible and are not:

• “Run everything” confuses effort with coverage. A giant suite still only tests the risks someone thought to encode; it says nothing about the new security surface a feature just opened, and it makes every change slow enough that the team starts skipping the gate — so it buys less real confidence over time, not more.
• “Test what’s easy” optimizes for the wrong quantity. Easy tests cluster where the code is simple, which is rarely where the risk is. You end up with 90% coverage of the boring 90% and zero coverage of the 10% that ships the outage.

A deliberate choice is the only one that spends the finite budget against actual risk. It is also honest: it makes visible, on the record, what you decided not to test and why — which is exactly the information you want when a change you deliberately under-tested turns out to bite. “We scored this low-risk and skipped the load test” is a decision you can learn from; “nobody thought about it” is not.

The architect’s lens

• Why does it exist? Because testing is finite spending, not an unlimited virtue — and without a method, effort defaults to easy or maximal, neither of which tracks risk. The selection method exists to point a limited budget at the risks that actually threaten this change.
• What problem does it solve? Misallocation: over-testing low-risk changes (waste) and under-testing high-risk ones (exposure). It converts a vague “did we test enough?” into a defensible, per-change decision.
• What are the trade-offs? It costs a few minutes of thinking up front, and it relies on judgment — a mis-scored risk means a mis-spent budget. It offers no automatic guarantee; it is a discipline, not a checklist you can outsource.
• When should I avoid it? Never skip the thinking, but do skip the ceremony for trivially safe changes — a help-text typo doesn’t need a written risk matrix; “read it, ship it” is itself a valid, scored decision. Don’t turn a lightweight method into heavyweight process.
• What breaks if I remove it? You fall back to the defaults. “Run everything” makes gates slow until people route around them; “test what’s easy” leaves the risky 10% naked. Either way, effort stops tracking risk — and the outage arrives from the surface nobody chose to look at.

Check your understanding

1. State the one idea of this page in a single sentence, and explain why over-testing and under-testing are the same underlying mistake.
2. List the four steps of the procedure in order, and say what each step produces.
3. A pure refactor (structure changes, behavior must not) and a dependency bump are different change kinds. Which risk class dominates each, and why do they differ despite both being “I changed code without adding a feature”?
4. The last step picks a type and a level together. Why is choosing the type alone insufficient? Give an example where the right type at the wrong level proves nothing.
5. Using the Knight Capital incident, explain how misclassifying a change — rather than “not testing enough” — is the root failure the selection method is designed to prevent.

Show answers

1. Testing is a finite budget spent to buy down risk, so you match the tests to the risk the change carries. Over-testing (heavy tests on a low-risk change) and under-testing (light tests on a high-risk one) are the same mistake — misallocation — spending budget somewhere other than where the risk actually is.
2. (1) Classify the change → its kind (bug fix, hotfix, feature, refactor, config, dependency bump). (2) Map → the risk classes that kind typically introduces. (3) Score → each risk as likelihood × cost on this change. (4) Pick → the type and level of the cheapest tests that cover the top-scored risks.
3. A refactor’s dominant risk is regression — behavior must be unchanged, so the whole job is proving nothing moved. A dependency bump’s dominant risks are regression and security — reused code meeting a new world (the Ariane 5 pattern) plus new CVEs the new version introduces or fixes. They differ because a refactor changes your code under your assumptions, while a bump changes someone else’s code and can shift assumptions you never wrote down.
4. Because type says what you look for and level says at what scope — both are needed to actually see a risk. Example: an auth-bypass is a security risk, but a security unit test can’t see it because the real middleware doesn’t run at unit scope; it needs at least integration level. Right type (security), wrong level (unit), zero confidence bought.
5. The deploy was treated as “just a config flag,” but it was really a config-plus-dormant-code-activation change across a fleet — whose characteristic risks are “did every node get the new state?” and “what old path does this flag wake up?” Those risks warranted deploy-verification and regression on the reactivated code. Correct classification is precisely what surfaces those risks; misclassifying the change routed the budget away from them, so the method’s first step — classify — is exactly the guard that was missing.