Example-Based vs Property-Based Testing

Every test you have written in this book so far — and almost every test written anywhere — is an example. You picked an input, you stated the expected output, and you asserted they matched: sort([3,1,2]) == [1,2,3]. The previous page, BDD and Gherkin, even dressed those examples in shared language — Given this input, When I call, Then expect that output. A Gherkin scenario is an example with a nicer suit on.

This page is the pivot of the Part. It names the one thing every example test has in common — you chose the input — and asks what happens when you stop choosing. What if, instead of asserting a specific output for a specific input, you assert a rule that must hold for every input in a domain, and let a tool generate the inputs for you? That is property-based testing, and the shift from “I pick the cases” to “the tool picks the cases” is the central tension the overview promised we would make precise here.

Two styles, defined from first principles

Both styles answer the same question — is this code correct? — but they encode “correct” in structurally different shapes.

Example-based testing pins a specific input to a specific expected output. You, the author, are the oracle: you compute (in your head, or by hand) what the answer should be for a case you chose, and you hard-code it.

  EXAMPLE-BASED
  ┌───────────────────────────────────────────┐
  │  input  YOU chose  ──►  output YOU assert   │
  │                                             │
  │  reverse("abc")  ==  "cba"                   │
  │  reverse("")     ==  ""                       │
  │  reverse("a")    ==  "a"                       │
  └───────────────────────────────────────────┘
  You are the oracle. Correctness = "these three
  pairs match." Coverage = exactly the pairs you typed.

Property-based testing states a rule — a property — that any correct output must obey, for all inputs drawn from a described domain. You do not name a single expected output. You name an invariant, and a generator supplies the inputs.

  PROPERTY-BASED
  ┌───────────────────────────────────────────┐
  │  ∀ s in Strings:                            │
  │      reverse(reverse(s))  ==  s              │
  │                                             │
  │  generator supplies s: "", "a", "abc",       │
  │  "🙂", "a\0b", 10_000-char noise, …          │
  └───────────────────────────────────────────┘
  The rule is the oracle. Correctness = "this holds
  for every s the generator threw." Coverage = the
  whole domain, sampled — including inputs you'd never type.

Notice what disappeared: in the property version, there is no hand-computed expected answer. reverse(reverse(s)) == s is true whether s is "abc" or a ten-thousand-character blob you would never sit and type. You did not have to know the answer for any particular input — you only had to know the rule.

Example tests have a structural limitation, and it is worth stating bluntly because it is the whole reason properties exist:

You only find the edge cases you already thought of.

An example suite is a list of inputs a human imagined. That list is bounded by that human’s imagination on the day they wrote it. The bug that takes down production is, almost by definition, the case nobody imagined — the empty list, the negative zero, the string with a null byte in the middle, the timestamp on the far side of the year 2038, the input of length exactly 2^31. If you had thought of it, you would have written the example and fixed the code. The dangerous inputs are precisely the ones absent from your list.

   the space of all inputs
   ┌─────────────────────────────────────────────┐
   │   ● ● ●   ← the ~12 examples you imagined     │
   │  ● ● ● ●                                       │
   │                                                │
   │                     ✗  ← the bug lives here,    │
   │                        in an input no human     │
   │                        on your team pictured    │
   └─────────────────────────────────────────────┘
   Example tests can only ever cover the dots.
   The ✗ is, by construction, somewhere else.

This is not a criticism of writing examples — it is a description of a ceiling. No amount of care raises it, because care operates on the inputs you conceive of, and the ceiling is the boundary of what you conceive of. To get past it you need a different source of inputs: not your imagination, but a generator that does not share your blind spots.

The invariant is the thing under test

The conceptual leap in property-based testing is that the property (or invariant) — not any single input/output pair — becomes the thing you assert. An invariant is a statement that stays true no matter which valid input you feed. Learning to see them is a skill; here are the recurring shapes, because most useful properties are an instance of one of these:

Round-trip (there-and-back). Encoding then decoding returns the original: decode(encode(x)) == x. Serialize/parse, compress/decompress, write-to-disk/read-from-disk. This is the single most productive property to reach for.
Invariance / oracle by construction. The output always satisfies a checkable structural rule: sort(xs) is always in non-decreasing order and is a permutation of xs. You never say what the sorted list is — only what must be true of it.
Commutativity / order independence. merge(a, b) == merge(b, a); adding items to a set in any order yields the same set.
Idempotence. Applying twice equals applying once: normalize(normalize(s)) == normalize(s); dedupe(dedupe(xs)) == dedupe(xs).
Model / equivalence. A fast implementation agrees with a slow, obviously-correct reference: myHashMap returns the same answers as a plain, unoptimized map for the same operations.

The most concrete one lives in this repo. The kvlite key-value store persists every write to a write-ahead log (WAL) and rebuilds its in-memory state by replaying that log on startup. That is a round-trip: whatever you write, then read back after a restart, must equal what you wrote.

// The example test the repo actually ships (paraphrased):
// specific keys, specific values, hand-chosen.
assert_eq!(round_trip(&mut r, &mut w, "SET name alice"), "OK");
assert_eq!(round_trip(&mut r, &mut w, "GET name"),        "alice");

// The PROPERTY hiding behind it — the invariant that must hold
// for ANY sequence of writes, not just "name → alice":
//
//   ∀ ops (a sequence of SET/DEL):
//     let db1 = apply(ops, fresh_db());
//     db1.flush();                 // write everything to the WAL
//     let db2 = Db::open(same_path);   // replay the WAL from scratch
//     assert_eq!(db1.snapshot(), db2.snapshot());
//
// "Anything I write and flush survives a restart, unchanged,
//  for every possible sequence of writes." A generator can
//  throw thousands of random op-sequences at that in one run.

The example test proves the WAL round-trips for name → alice. The property proves it round-trips for every sequence of operations a generator can invent — including the sequence with an empty value, the key containing a newline, the DEL of a key that was never SET, the 50,000-operation torrent. The invariant is one line; the inputs are unbounded.

Suppose you want to be confident that parse(format(n)) == n for a signed 32-bit integer formatter/parser pair.

  Domain size (all i32):        2^32  ≈  4.29 × 10^9 values
  Examples a diligent dev writes: ~15 (0, 1, -1, MIN, MAX, a few random)
  Fraction pinned by examples:    15 / 4.29e9 ≈ 0.0000000035 %

  One property, run for 1,000 generated cases per CI run,
  across (say) 500 CI runs over a year:
      500 × 1,000  =  500,000 distinct values exercised,
      biased toward 0, ±1, MIN, MAX — the boundaries where
      off-by-one and overflow bugs actually live.

  Lines of test code:   examples ≈ 15+     property ≈ 3

The property is shorter to write and covers ~30,000× more of the space, and — crucially — it keeps finding new inputs on every run instead of re-checking the same fifteen forever.

Under the hood — why “assert a rule” changes what you can express

The deep difference is not tooling — it is what kind of claim each style can even state. An example test can only assert extensional facts: “for this exact input, the output is this exact value.” It talks about points. A property asserts an intensional fact: a relationship the output must bear to the input, or to itself, regardless of value. It talks about laws.

That is why some correct properties are things you could not write as an example even if you tried. Consider sort:

  As examples, you can only ever say:
      sort([3,1,2]) == [1,2,3]        (a point)
      sort([])      == []             (a point)
      … one hand-computed answer per input …

  As a property, you say the LAW that defines "sorted":
      ∀ xs:  isNonDecreasing(sort(xs))         -- structural rule
      ∀ xs:  isPermutation(sort(xs), xs)       -- nothing lost/added

  Note you never had to KNOW the sorted answer for any xs.
  The two rules together fully pin "correct sort" — for a
  MILLION inputs you never computed by hand.

This is also the trap: because a property does not name an output, a weak property can be true of a broken function. length(sort(xs)) == length(xs) is a real property — but a sort that returns its input unchanged passes it. A good property constrains the output enough that only correct implementations satisfy it (the two sort rules above do; the length rule alone does not). Writing that constraint well is the actual skill, and it is the subject of the next page.

Complementary, not rivals

It is tempting to read all of this as “properties are better, delete your examples.” That is the wrong lesson, and getting it wrong produces worse test suites, not better ones. The two styles do different jobs and belong together.

Keep example tests for what they are unmatched at:

Documentation. parseAmount("$1,000.50") == 100050 tells a reader, at a glance, exactly what the function does and in what units. A property (parse and format are inverses) is precise but abstract; the example is legible. New teammates learn the API from examples, not from ∀.
Pinning known cases. Every bug you fix should leave behind an example test for the exact input that broke — the regression test. That is a specific, named case you want frozen forever, not re-rolled by a generator.
The cases you were explicitly asked for. Acceptance criteria and BDD scenarios (page 5) are examples on purpose: they encode a stakeholder’s concrete “when I do X I expect Y.”

Add property tests for what examples cannot do:

Hunting unknown unknowns. The generator explores the region your imagination can’t reach — that is the entire point.
Stating intent at the level of rules. “Output is always sorted” is a stronger, more honest claim than “these six lists came out sorted.”

   ┌──────────────── a healthy test suite ────────────────┐
   │                                                        │
   │  EXAMPLES  →  readable spec, regressions, asked-for     │
   │               cases.  "Here is what it does."           │
   │                                                        │
   │  PROPERTIES →  invariants over the whole domain.        │
   │               "Here is what must ALWAYS be true —       │
   │                now go try to break it, machine."        │
   │                                                        │
   └────────────────────────────────────────────────────────┘
   Examples say what. Properties say what-must-hold.
   You want both sentences on the record.

The mental model: an example is a point, a property is a shape. You want a few well-chosen points to make the behavior legible, and a shape to guarantee the space between and beyond those points behaves. Neither replaces the other.

In 1994, Intel’s Pentium processor was found to return subtly wrong results for certain floating-point divisions — a lookup table used by the division algorithm was missing a handful of entries, so a small, hard-to-hit set of operand pairs produced errors in the later digits. Intel initially downplayed it; the eventual recall and replacement program is widely reported to have cost on the order of several hundred million dollars.

The testing lesson is the one this page is built on. The faulty operands were a sparse, unremarkable-looking subset of an astronomically large input space — exactly the kind of case no hand-written example list would happen to include, because nothing about those particular numbers looks special. A property stated over division — for instance, “(a / b) * b is within floating-point tolerance of a for all a, b” — is precisely the kind of invariant a generator hammers with thousands of operand pairs, and it does not need to know in advance which pairs are dangerous. That is the shift: examples require you to guess where the bug is; a property lets the machine go looking. (Figures are the widely-cited public estimates; the lesson, not the exact dollar amount, is the point.)

What makes this practical — the next two pages

A fair objection: “if the generator invents a 50,000-operation sequence and it fails, I now have a 50,000-operation failure to debug — that’s worse than a hand-written example, not better.” That objection is correct, and it is exactly why property-based testing needed two ideas to become usable in practice, which the next two pages supply:

Generators — how a tool produces valid, well-distributed inputs for your domain, and how you write invariants that catch real bugs (rather than lazy ones that always pass). That is Property-Based Testing: Generators and Invariants.
Shrinking — the superpower that answers the objection above. When a property fails on that 50,000-operation monster, the tool automatically shrinks it to the minimal failing input — often two or three operations — handing you the smallest possible reproduction. That is Shrinking and the Edge-Case Superpower.

Generators make the search possible; shrinking makes the failures debuggable. Together they turn “assert a rule over all inputs” from a nice idea into a technique you run in CI.

The architect’s lens

Why does it exist? Because example-based testing has a hard ceiling — it can only ever check inputs a human imagined — and the bugs that reach production live, by construction, in the inputs nobody imagined. Property-based testing exists to source inputs from a generator instead of from your imagination.
What problem does it solve? The blind-spot problem: it hunts the unknown unknowns. By asserting an invariant over a whole domain, it lets a machine explore the space between and beyond your hand-picked points and surface edges (-0.0, empty, null byte, boundary integers) you would never type.
What are the trade-offs? You give up the legible, self-documenting “input → output” pair, and you take on real difficulty: writing a good invariant is hard, and a weak one (result != null) gives false comfort while always passing. Failures also arrive as generated inputs you must interpret (mitigated by shrinking).
When should I avoid it? When there is no invariant cheaper to state than the implementation itself (much straight-line business logic), when you need a readable spec or a stakeholder-facing acceptance case, or when pinning one specific regression — reach for an example there. Properties are a supplement, never a replacement.
What breaks if I remove it? You keep only the inputs you thought of, so your confidence silently caps at the edge of your imagination. Whole classes of boundary and round-trip bugs stay invisible until a user — or an Ariane 5, or a Pentium — finds them for you in production.

Check your understanding

In one sentence each, define example-based and property-based testing, framed around who chooses the input.
State the structural “blind spot” of example tests in your own words. Why can’t more diligence or more examples raise that ceiling?
What is an invariant, and why is it — rather than any specific expected output — “the thing under test” in a property? Give the round-trip invariant for the kvlite WAL.
This page argues examples and properties are complementary, not rivals. Name two jobs an example test does that a property cannot, and one job a property does that examples cannot.
Someone says “property-based testing is worse because a failure is a giant random input I can’t debug.” Which upcoming technique answers this objection, and how?

Show answers

Example-based: you choose a specific input and assert a specific expected output (reverse("abc") == "cba"). Property-based: you state a rule that must hold for all inputs in a domain, and a generator chooses the inputs (∀ s: reverse(reverse(s)) == s). The pivot is who picks the input — you, versus the tool.
Example tests can only check inputs a human imagined, so their coverage is bounded by that imagination; the production-breaking bug is, almost by definition, an input nobody pictured. More diligence just adds more of the inputs you can conceive — it operates below the same ceiling, because the ceiling is the edge of what you conceive.
An invariant is a statement that stays true for every valid input (e.g. “the output of sort is non-decreasing and a permutation of the input”). It is the thing under test because a property asserts the rule, not a hand-computed answer, so it can be checked against inputs whose “correct” output you never computed. The kvlite WAL invariant: for any sequence of writes, what you read back after flushing and replaying the log on restart equals what you wrote — decode(encode(x)) == x.
Examples uniquely: document the API legibly (parseAmount("$1,000.50") == 100050 shows units at a glance), pin known regressions to an exact input, and encode explicitly-asked-for acceptance cases. Properties uniquely: hunt unknown unknowns by exploring the domain a human can’t enumerate (and state intent at the level of rules, e.g. “always sorted”).
Shrinking (shrinking-and-edge-cases). When a property fails on a large generated input, the tool automatically reduces it to the minimal failing case — often two or three elements — so you debug the smallest reproduction, not the giant random one the generator first found.