Revision — Types of Testing

The previous part sliced testing by level — how much of the system a test puts under test, from a single function to the whole running product. This part cut across that axis with a second question: not how wide you look, but what you are looking for. That is the type axis, and the whole part was a tour of the types worth knowing and, more importantly, the specific risk each one exists to buy down.

This page walks the ground once more, quickly, so the shape stays in your hands after the detail fades. No new material — just the map, redrawn.

The organizing split

Everything in this part hangs off one line: functional versus non-functional.

Functional testing asks does the software do the right thing? — the right output for a given input, the right behaviour against the specification. It is a pure question of correctness, blind to how fast, how secure, or how pleasant the software is. A functional bug is a wrong answer, and it fails loudly: an assertion you can write flips from pass to fail.

Non-functional testing asks does it do it well enough? — the qualities of behaviour that is already correct. Speed, resilience, security, usability, accessibility, compatibility: the “-ilities.” A non-functional bug is a right answer delivered badly — the total is exact but arrives eight seconds late, or leaks a session token, or renders as invisible grey-on-white for a colour-blind user. It sails past every functional test, because the functional test never asks the question.

                     does it work?
                          │
        ┌─────────────────┴──────────────────┐
   FUNCTIONAL                            NON-FUNCTIONAL
"does it do the right thing?"       "does it do it well enough?"
        │                                     │
 a wrong answer                    a right answer delivered badly

Two things are worth restating about this split. First, the two sides fail differently and are found differently — which is exactly why you need tools on both. Second, type cuts across level. “Unit” is a level; “performance” is a type. You can run a performance test as a micro-benchmark on one function or as a load test on the whole service. You can run a functional test at any level too. The two axes are orthogonal: level tells you how wide you look, type tells you what you are hunting. Confusing them is one of the most common mistakes in how people talk about testing.

Each type, and the risk it buys down

The unifying idea of the part: a type of testing is a lens ground for one class of risk. You do not run “all the types” any more than a doctor orders every possible scan. You run the ones that address the risks actually present in the change in front of you. Here is the whole catalogue, each in one line, paired with the risk it addresses.

Functional testing buys down wrong-behaviour risk — the software computes, returns, or decides the wrong thing for a given input. It is the type every change touches first: if it does the wrong thing, nothing else matters.
Regression, smoke & sanity buy down the risk that a change silently breaks something that already worked. Regression is the broad net cast over past behaviour; smoke is the fast “is it even alive?” gate on a fresh build; sanity is the narrow “did the thing I just touched actually work?” spot-check. Different widths, same fear — that fixing here broke there.
Performance testing buys down “correct but too slow, or collapses under load” risk. Load asks how does it behave at expected traffic?, stress where does it break?, spike can it survive a sudden surge?, and soak does it degrade — leak memory, fragment — over hours or days? Four questions, one worry: the right answer that arrives too late or not at all.
Security testing buys down “correct but exploitable” risk — data or access an attacker can take that the specification never meant to give. It is the type you reach for when a change touches input, authentication, secrets, or a trust boundary, and it demands you stop thinking like a user and start thinking like an attacker.
Accessibility, usability & compatibility buy down “correct but unusable” risk — for some users, on some devices, in some browsers. Accessibility asks can everyone operate it, usability can they operate it easily, compatibility does it work outside the one environment you built it in. The answer can be perfectly right and still shut real people out.
Exploratory testing buys down unknown-unknowns risk — failures on the paths no script ever imagined. It is not the absence of rigour; it is simultaneous learning, test design, and execution, steered by a tester’s judgement toward the places a written suite structurally cannot reach.

Notice the pattern: no type is “best,” and none is a competitor to another. A functional test and a security test are not rivals — they answer different questions, and a change can need both, one, or neither. Each catches a class of failure the others structurally cannot. That is the type-level echo of the levels-part lesson.

Two axes, one grid

It is worth pausing on how this part sits beside the previous one, because the whole point of keeping them separate is that together they form a grid, not a list. The levels part gave you a vertical axis — unit, integration, system, end-to-end — answering how much of the software runs during the test. This part gave you a horizontal axis — functional, regression, performance, security, accessibility, exploratory — answering what quality of the software you are checking. Any real test occupies one cell of that grid: it is a type run at a level.

                unit ── integration ── system ── end-to-end   (LEVEL: how wide)
              ┌──────────────────────────────────────────
   functional │  x        x            x          x
   regression │           x            x          x
   performance│  x(bench)               x(load)
   security   │  x        x            x          x
   a11y/usab. │           x(component)            x
   exploratory│                                   x (by hand)
        (TYPE: what for)

The grid is why “run more tests” is the wrong instinct and “run the right cell” is the right one. A performance concern is not served by adding unit tests; a functional bug is not served by a load run. When you name a change’s risks, each risk points at a type (a column), and then the risk’s nature points at a level (a row). That intersection is the test you actually write. Holding both axes at once is the muscle this part and the last one were built to grow.

The closing skill: read the change, pick the types

The first six pages each hand you one lens. The seventh — Choosing What a Change Needs — is where the part pays off, because it turns the catalogue into a decision. A test costs time to write, run, and maintain; you cannot run every type on every change and no one has that budget. So before a single assertion, you ask: what could go wrong with this change, and which of those failures do I most need to rule out?

That is a question about risk, and risk is roughly likelihood × cost. You score each class of failure — how likely on this change, times what it costs if it escapes — and you spend where that product is highest, not evenly.

  a change ships some risk ──► pick the test type that buys down THAT risk
                          ──► then pick the level that risk lives at

  "computes the wrong answer"      ─► functional        (unit → system)
  "my fix broke something else"    ─► regression/smoke   (integration → system)
  "correct but too slow"           ─► performance        (micro-bench → load)
  "correct, but an attacker gets in"─► security          (unit → system, + pentest)
  "correct, real users can't use it"─► a11y/usability    (component → e2e)
  "correct on the paths I imagined"─► exploratory        (whole system, by hand)

The skill is two moves, not one. First, type: read the diff and name its risks — a typo fix in help text carries almost none, while a change to how passwords are hashed carries functional, regression, security, and performance risk all at once. Same tool-belt, wildly different selections, because the risk profiles differ. Second, level: for each type you chose, decide where it belongs — a functional check might be a unit test or an end-to-end test; a performance concern might be a micro-benchmark or a full load run. Tying type to level — deciding not only what to test but where that test lives — is what turns a list of testing words into a concrete plan for one specific change.

The method, walked once

To see both moves working together, walk one change end to end — the same shape the decision-guide page taught. Take a diff that adds a coupon field to a checkout form. The method is not “run everything”; it is read, name, score, then choose type-and-level.

  change: "add a coupon field to checkout"

  1. READ   — what does this touch? a form input, the order-total path,
              user-supplied data flowing into a money calculation.

  2. NAME   — which risks ride along?
              functional   (the total could come out wrong)
              regression   (the total path is shared by many features)
              security     (user input into a price → injection/abuse)
              accessibility(a new field real users must operate)
              performance  (one field on a hot path — probably minor)

  3. SCORE  — likelihood × cost, per risk:
              wrong total .............. likely × high  ► YES
              broke the order path ..... likely × high  ► YES
              injectable coupon ........ possible × very high ► YES
              unlabeled field .......... likely × medium ► YES
              slower checkout .......... unlikely × medium ► maybe

  4. CHOOSE type AND level:
              functional  → unit tests on the total calc + one e2e on the flow
              regression  → run the existing order-total suite
              security    → validation + injection tests at the input boundary
              a11y        → label + keyboard-operability check on the component
              performance → skip unless it adds a DB round-trip

Notice what the method buys you: not just which tests, but a defensible no to the ones you skipped. “I didn’t load-test a single form field” is a decision you can state and justify — which is exactly what “justified confidence” means when someone asks why you shipped.

The same walk on a different change lands somewhere else entirely. A one-word fix to a help-text string carries almost no risk of any kind; reading it and shipping it, with maybe a smoke check that the page still renders, is a legitimate testing decision — a load test on a string constant would be absurd. Same method, opposite answer, because the risk profile is opposite. The method is the constant; the selection is what varies.

On the morning of 1 August 2012, the trading firm Knight Capital deployed new order-routing software to its production servers. By the firm’s own account of the episode, the deployment did not reach every server, and old code left on the machines that missed the update was reactivated by a repurposed flag — so some servers ran new logic while others ran stale logic. Within roughly the first hour of trading the system fired millions of unintended orders into the market. The firm has been reported to have lost on the order of hundreds of millions of dollars, an amount that pushed it to the edge of collapse within days.

Read it through this part’s lens. Every individual order the software sent may have been functionally correct in isolation — this was not one wrong answer. The failures that mattered were the ones the functional path never asked about: a regression/deployment failure (a change that silently reactivated behaviour everyone assumed was dead) and, underneath it, the absence of a fast smoke/sanity gate that would have caught “these servers are not running what we think they are” before real money flowed. It is the clearest possible argument for why type cuts across level and why non-functional risk deserves its own lens: a suite that only checks does each order compute correctly? passes clean while the firm burns. Specifics here are widely reported but hedge them if you cite them; the lesson does not depend on the exact figure.

The throughline, restated

Every page in this book serves one question: how do you earn justified confidence that software works — and how do you think about the edge cases that break it? This part answered the first half by refusing to treat confidence as a single thing. Confidence that it computes the right answer is not confidence that it stays fast, or stays safe, or stays usable. Each is a separate lens, earned separately, and “justified” means you deliberately chose the lenses the change’s risk profile warranted — and can say why you skipped the rest.

It answered the second half through exploratory testing, the part’s discovery engine. Scripted tests confirm the edge cases you already thought of; exploratory testing is how you find the ones you didn’t — the unknown unknowns that no assertion was ever written for. Hunting those edge cases on purpose, rather than waiting for production to find them for you, is the habit the whole book is trying to build.

So carry two sentences out of this part. A type of testing is a lens ground for one class of risk — you reach for the lens the change in front of you actually needs, at the level that risk lives. And: confidence is earned per-risk, deliberately, with the edges hunted rather than hoped away. With levels and types both in hand — the two axes of how wide and what for — you now have the full grid for deciding what any change needs. The next part builds on exactly that grid, turning these choices into a repeatable practice you run on every change you ship.