Testing ML Systems — When There Is No Fixed Oracle
The last few pages let a machine search for the inputs that break code: fuzzing mutated bytes to find crashes, property testing generated typed inputs to find wrong answers, and chaos engineering broke the environment to find fragile assumptions. Every one of those techniques still leaned on an oracle — some notion of “correct” you could name in advance, even if that notion was as thin as “did not crash.”
This page removes that crutch. A machine-learning model is the case where you genuinely cannot write down the expected output ahead of time. Ask a sentiment classifier about a sarcastic review, a translation model about an idiom, or an image model to describe a photo, and there is no single string you can assert_equal against. The right answer is a fuzzy region, not a point — and the model may return slightly different points on different runs. The whole discipline of testing has to be re-derived for systems with no fixed oracle, and that re-derivation is this page.
Why the classic test breaks
Section titled “Why the classic test breaks”The unit test you have relied on all book has a shape: call the function with a fixed input, assert it equals a fixed expected output.
THE CLASSIC TEST WHY IT FAILS ON AN ML MODEL ──────────────── ─────────────────────────── out = f(input) out = model(input) assert out == EXPECTED assert out == EXPECTED ← what IS expected? • no single correct output (fuzzy target) • non-deterministic (temperature, GPU, batch order, float non-associativity) • "close enough" ≠ "equal"Two independent things break the equality.
- There is no single correct output. Recall the oracle problem from Part 1: an oracle is whatever tells you an output is right. For most ML tasks there is no such oracle you can encode. “Translate this sentence” has thousands of acceptable outputs; “summarize this article” has infinitely many. Even a human panel would disagree. Hard-coding one expected string tests your opinion of one phrasing, not the model’s correctness — and it will fail the moment the model retrains and picks a different, equally good phrasing.
- The output is non-deterministic. The same input can produce different outputs across runs. Sampling with non-zero temperature is the obvious source, but even a “deterministic” model drifts: floating-point addition is not associative, so a different GPU, a different batch size, a different library version, or a reordered reduction can change the last bits of a score and flip a borderline classification. The determinism controls from Part 6 — pin the seed, freeze the clock — help, but they cannot make a fuzzy target sharp.
So assert_equal on an exact value is the wrong tool twice over. You need tests that do not require you to name the answer.
Metamorphic testing — check a relation, not an output
Section titled “Metamorphic testing — check a relation, not an output”The key technique for a system with no oracle is metamorphic testing. You already met the seed of it as a “metamorphic property” in the fuzzing-vs-property-testing page: when you cannot state the exact right answer for a single run, state a relation between the outputs of two related runs.
The insight: you may not know model(x), but you often know how model(x) and model(x') must relate when x' is a controlled transformation of x. That relation is testable without ever naming a correct output.
transform T x ─────────────────► x' │ │ model│ │model ▼ ▼ y = model(x) y' = model(x')
METAMORPHIC RELATION: some known relation R(y, y') must hold — even though you cannot say what y or y' individually "should" be.Three concrete metamorphic relations you can write today:
- Paraphrase / perturbation invariance. Rephrase the input in a meaning-preserving way — swap a word for a synonym, add a trailing space, change “don’t” to “do not” — and the label should not change.
classify("This film was great") == classify("This movie was great"). You never assert which label; you assert the two agree. - Translation round-trip. Translate English→French→English. The result is not identical to the original (there is no oracle for the round-trip either), but it should stay close in meaning — high semantic similarity, same named entities, same numbers. The relation is “round-trip preserves meaning,” not “round-trip is the identity.”
- Monotonicity under a scaled feature. For a model with a directional feature, increasing that feature should move the prediction the expected way. A loan-risk model’s predicted risk should not decrease when you raise debt while holding everything else fixed; a price model’s estimate should not drop when you add a bedroom. You assert the direction of change, never the value.
RELATION TRANSFORM T ASSERTION (no exact output) ───────── ─────────── ─────────────────────────── invariance synonym / whitespace / casing label(x') == label(x) round-trip en→fr→en sim(round_trip(x), x) ≥ τ monotonicity raise feature f, hold rest pred(x') ≥ pred(x) consistency reorder a set-valued input output unchangedHere is the same idea as a Rust-flavoured test skeleton, in the property-test style the book has used since Part 5 — note there is no expected label anywhere in it:
// Metamorphic: a meaning-preserving paraphrase must not flip the label.proptest! { #[test] fn paraphrase_is_label_invariant(review in review_strategy()) { let original = classify(&review.text); let paraphrased = classify(¶phrase(&review.text)); // synonym swap, etc. // We never say WHICH label is correct — only that T preserves it. prop_assert_eq!(original.label, paraphrased.label); }}The generator invents reviews; the transform paraphrases them; the assertion is a relation. When it fails, the shrunk counterexample is a minimal input pair where a trivial rewording flipped the model’s mind — a genuine robustness bug you found without an oracle.
Property and invariant testing for models
Section titled “Property and invariant testing for models”Metamorphic relations are one family of ML properties. There are others you can assert about a single run, no paired input required — the same property-based mindset from earlier Parts, pointed at a probabilistic function.
- Robustness to small perturbations. For an input
xand a tiny perturbationδ(imperceptible noise on an image, a typo in text), the prediction should not change:pred(x + δ) == pred(x)for‖δ‖below some bound. A violation is an adversarial example — a real failure class, not a curiosity. - Fairness / group constraints. Across protected groups, a chosen fairness metric must stay within a tolerance. For example, the positive-prediction rate for group A and group B should differ by no more than
ε(demographic parity), or error rates should be comparable across groups. You are not asserting any one person’s outcome; you are asserting a statistical relation across a slice. - Bounded-output sanity checks. The cheapest, most under-used ML tests. A probability must be in
[0, 1]; a softmax vector must sum to 1; a predicted age must be non-negative; a recommended quantity must not exceed stock; a generated summary must not be longer than its source. These catch broken pipelines, bad post-processing, and numerical blowups instantly — and they do have a fixed oracle, so write them as ordinary asserts.
INVARIANT CHECK CATCHES ───────── ───── ─────── robustness pred(x+δ) == pred(x), ‖δ‖ small adversarial fragility fairness |rate(A) − rate(B)| ≤ ε disparate impact bounded output 0 ≤ p ≤ 1, Σ softmax == 1 broken post-processing monotonic feature raising f never lowers risk nonsensical logicNote the split: bounded-output checks are deterministic asserts you should have anyway; robustness and fairness are statistical claims you check over many inputs, closer to a threshold than an equality.
Evaluation as testing on distributions
Section titled “Evaluation as testing on distributions”Everything above tests one input or one pair at a time. But the headline claim about an ML model — “it is good enough to ship” — is a claim about many inputs at once. That is what evaluation is: testing on a distribution instead of an example.
PER-EXAMPLE TEST DISTRIBUTIONAL TEST (evaluation) ──────────────── ──────────────────────────────── assert model(x) == y run model over a whole TEST SET pass / fail on one input compute a METRIC (accuracy, F1, BLEU, ...) pass iff metric ≥ ACCEPTANCE THRESHOLDThree pieces make this a real test, not a vibe:
- A held-out test set. A fixed, versioned dataset the model never trained on — the ML analogue of a regression suite. It must stay held out: if it leaks into training, your test grades the model on its own answer key, and the number is a lie.
- A metric with an acceptance threshold. Pick a metric appropriate to the task (accuracy, precision/recall, F1, AUC, BLEU/ROUGE for text, calibration error) and a gate: “F1 on the test set must be ≥ 0.90, and must not drop more than 1 point versus the current production model.” Now “does it pass?” has a crisp answer again — just at the population level, not the example level.
- Slice-based evaluation. An aggregate number hides the failures that matter. A model at 95% overall can be at 60% on a minority slice — a language, a device type, a rare class, a demographic group. So you evaluate per slice and gate each one, not just the average. This is where per-example pass/fail is replaced by per-slice pass/fail.
OVERALL ACCURACY: 95% ← looks shippable ───────────────────────────────────────── slice: en 97% ✓ (threshold 90%) slice: es 96% ✓ slice: low-light 62% ✗ ← the aggregate hid this slice: age > 65 71% ✗ ───────────────────────────────────────── GATE = every slice ≥ 90% → FAIL, even though the mean passed.The mental shift: a green suite for an ML system is not “N assertions passed.” It is “every metric cleared its threshold on every slice of a held-out distribution.”
Testing the pipeline, not just the model
Section titled “Testing the pipeline, not just the model”A subtle trap: teams pour effort into evaluating the model and forget that the model is one component in a pipeline — data in, features, training, serving — and most production ML failures are pipeline failures, not model failures. The weights can be perfect while the system is broken.
raw data ─► schema/validation ─► features ─► TRAIN ─► model ─► SERVE ─► prediction │ │ │ │ │ │ └── same feature code here ───┘ (or SKEW) │ └── types, ranges, nulls, cardinality drift └── the thing that silently poisons everything downstreamThree tests that live outside the model:
- Data-schema validation. Before training or serving, assert the incoming data matches an expected schema: column types, value ranges, allowed categories, null rates, cardinality. A feature that was
0–1yesterday and arrives as0–100today (someone changed units upstream) will silently wreck predictions. This is an ordinary assertion with a fixed oracle — the schema is the oracle — and it is the highest-return ML test there is. - Training/serving skew. The features computed at training time must be computed the same way at serving time. If training uses a batch job that fills missing values with the column mean but serving fills them with zero, the model sees inputs it never trained on and quietly degrades. The test: run the same input through both feature paths and assert they produce identical features.
- Reproducibility of the training run. Pin the data version, the code version, the random seed, and the library versions, then assert that re-running training produces the same model (or metrics within a tiny tolerance). If you cannot reproduce a training run, you cannot debug a regression, bisect a quality drop, or trust that “we retrained and it got better” was causal rather than noise.
Under the hood — why “justified confidence” changes shape
Section titled “Under the hood — why “justified confidence” changes shape”The book’s throughline is justified confidence that software works. For deterministic code, justified confidence is deductive-ish: a green suite of exact assertions says “for these inputs, the output is provably this.” For probabilistic systems, that guarantee is not available, so the shape of the confidence changes from deterministic to statistical.
DETERMINISTIC SYSTEM PROBABILISTIC (ML) SYSTEM ──────────────────── ───────────────────────── assert output == expected assert metric ≥ threshold, per slice pass = every case correct pass = quality holds across a distribution confidence: "this input is right" confidence: "this population is good enough, and these relations always hold"You are not claiming any single prediction is correct — you cannot. You are claiming three weaker-but-honest things: (1) a set of relations always holds (metamorphic relations and invariants — these are hard assertions), (2) aggregate quality clears a threshold on a held-out distribution, and (3) that quality holds on every slice you care about, not just on average. Justified confidence for an ML system is a portfolio of statistical guarantees over slices plus a handful of deterministic invariants — never a wall of exact-match assertions. When the world later shifts under that distribution, no offline test can catch it — which is why the next page moves to data drift and production monitoring.
The architect’s lens
Section titled “The architect’s lens”- Why does it exist? Because a machine-learning model has no fixed oracle: for most tasks there is no single correct output to assert against, and the output is non-deterministic, so the classic
assert output == expectedunit test simply has nothing valid to write. - What problem does it solve? It restores testability to systems with fuzzy, shifting targets — by checking relations between related runs (metamorphic), invariants (robustness, fairness, bounded outputs), and aggregate quality on distributions (metrics with thresholds, per slice) instead of exact per-example equality.
- What are the trade-offs? You trade a crisp deductive pass/fail for a statistical one: tests are probabilistic, thresholds are judgement calls, evaluation needs a curated held-out set, and a metamorphic relation only tests the relation you thought of — never full correctness.
- When should I avoid it? When the component actually has a fixed oracle — a bounded-output check, a schema, a deterministic feature transform. Those deserve ordinary exact assertions; don’t reach for statistical machinery where a plain
assertis correct and stronger. - What breaks if I remove it? You fall back to hard-coding one “expected” answer per input — brittle tests that break on every retrain and prove nothing about correctness — or you ship on gut feel, letting fairness gaps, adversarial fragility, and training/serving skew reach production untested.
Check your understanding
Section titled “Check your understanding”- Give the two independent reasons
assert output == expectedfails for a machine-learning model, and name a source of non-determinism that persists even when you set temperature to zero. - Explain metamorphic testing in one sentence, then give two concrete metamorphic relations you could assert without ever naming a correct output.
- Bounded-output checks (a probability in
[0,1], a softmax summing to 1) are ML tests too — but they differ from robustness and fairness checks in a fundamental way. What is that difference, and why does it mean you write them as ordinary asserts? - Why can an ML model with 95% overall accuracy still be unshippable, and what evaluation practice is designed to catch that? Tie your answer to the dated incident on this page.
- “Justified confidence” for a probabilistic system has a different shape than for deterministic code. Describe that shape in terms of relations, distributions, and slices — and say why it is honest rather than a weakening of standards.
Show answers
- First, there is no single correct output — most ML tasks (translate, summarize, classify sarcasm) have a fuzzy region of acceptable answers, not one string, so there is nothing valid to hard-code as
expected. Second, the output is non-deterministic — the same input can produce different outputs across runs. A source that persists at temperature zero: floating-point non-associativity, so a different GPU, batch size, or library version can change the last bits of a score and flip a borderline result. - Metamorphic testing checks a known relation between the outputs of two related runs (an input and a controlled transformation of it) when you cannot state the correct output for either run. Two relations: paraphrase invariance — a synonym swap must not change the label (
classify(x) == classify(paraphrase(x))); and monotonicity — raising a directional feature while holding the rest fixed must move the prediction the expected way (e.g. more debt must not lower predicted loan risk). (Translation round-trip preserving meaning is also acceptable.) - Bounded-output checks have a fixed oracle — the rule itself (probability ∈
[0,1], softmax sums to 1) is the correct answer, deterministically true for every valid output — so you write them as ordinary exactasserts. Robustness and fairness checks are statistical claims with no single correct output; you verify them over many inputs against a tolerance/threshold, not as one exact equality. - Because an aggregate metric hides slice-level failures: a model at 95% overall can be at 60% on a minority slice (a language, device type, rare class, or demographic group). Slice-based evaluation — measuring the metric per slice and gating each one, not just the average — is designed to catch it. The dated incident: audits (around 2018) found commercial face-analysis systems with far higher error rates on darker-skinned and female faces while overall accuracy looked strong; only per-slice evaluation and a fairness invariant would have surfaced and blocked it.
- For deterministic code, confidence is per-example and near-deductive: a green suite of exact assertions says “these inputs give provably these outputs.” For a probabilistic system it becomes statistical: (1) a set of relations always holds (metamorphic relations and invariants — themselves hard assertions), (2) aggregate quality clears a threshold on a held-out distribution, and (3) that quality holds on every slice you care about, not just on average. It is honest, not weaker, because it claims exactly what is true — you cannot prove any single fuzzy prediction “correct,” so you make the strongest verifiable claims available: guaranteed relations plus thresholded quality over slices.