Data-Driven and Parameterized Tests

The page object patterns we just met were about not duplicating structure — one place that knows what a screen looks like. This page is about the other great duplication in an automated suite: not duplicating the test body when the only thing that changes from one case to the next is the data.

Almost every technique from the tester-mindset and edge-cases Parts produces the same shape of work: for input X, expect output Y — and then a dozen more (X, Y) pairs at the boundaries. If you write each of those as its own copy-pasted test function, you have built a maintenance liability and, worse, a place for coverage gaps to hide in plain sight. Parameterization is the mechanism that turns that dozen pairs into one test body and a table of rows. It is how boundary-value and equivalence-class analysis become cheap enough to actually run.

The problem: copy-paste is a coverage risk, not just ugly

Suppose you are testing a function that validates a username. The rules give you a handful of cases: too short, too long, valid, contains a space, empty. The naive suite looks like this:

def test_username_too_short():
    assert is_valid("ab") is False

def test_username_ok():
    assert is_valid("ada") is True

def test_username_too_long():
    assert is_valid("a" * 33) is False

def test_username_with_space():
    assert is_valid("ada lovelace") is False

# ...and the empty-string case? Did anyone write it?

Every function is a near-identical clone: same call, same assertion shape, only the literals differ. Three problems follow directly, and none of them is cosmetic.

• Maintenance liability. Change the assertion style — add a message, switch is_valid to return a reason — and you edit it in five places. Miss one and the suite is quietly inconsistent.
• Hidden gaps. Because each case is a separate hand-written function, the absence of a case is invisible. Nothing in the file announces “the empty-string boundary is untested.” A missing row in a table is a visible hole; a missing function in a wall of similar functions is not.
• Drift. Copy-paste invites copy-paste error: one test still asserts True where it should assert False because you edited the input but forgot the expectation. The tests agree with each other’s shape and disagree with the spec, and they still pass.

The fix is to separate the thing that varies (the data) from the thing that stays the same (the logic).

   copy-pasted tests                 parameterized test
   ─────────────────                 ──────────────────
   test_a: call() assert x           ┌── one test body: call() assert expected
   test_b: call() assert y           │
   test_c: call() assert z           └── table of rows:
   test_d: call() assert w                 (a, x)
   ...                                      (b, y)
   logic duplicated N times                (c, z)  ← add a row, not a function
                                            (d, w)

Parameterized and table-driven tests

A parameterized test (also called table-driven or data-driven) is one test body executed once per row in a table of cases. Each row is typically a tuple of (input…, expected), sometimes with a name. The pattern is old and universal — every serious framework has it — because it matches the real structure of the work: the logic is written once, the cases are enumerated as data.

Here it is in three common flavours. First, Python with pytest:

import pytest

@pytest.mark.parametrize(
    "name,        raw,             expected",
    [
        ("too_short",  "ab",            False),
        ("minimum_ok", "abc",           True),   # boundary: 3 is the first valid length
        ("maximum_ok", "a" * 32,        True),   # boundary: 32 is the last valid length
        ("too_long",   "a" * 33,        False),
        ("has_space",  "ada lovelace",  False),
        ("empty",      "",              False),  # now the gap is a *visible row*
    ],
)
def test_is_valid(name, raw, expected):
    assert is_valid(raw) is expected, f"case {name!r}: is_valid({raw!r})"

Go builds the table by hand — this is where “table-driven” got its name — and the idiom is so standard it is in the language’s own documentation:

func TestIsValid(t *testing.T) {
    cases := []struct {
        name string
        raw  string
        want bool
    }{
        {"too_short", "ab", false},
        {"minimum_ok", "abc", true},
        {"maximum_ok", strings.Repeat("a", 32), true},
        {"too_long", strings.Repeat("a", 33), false},
        {"empty", "", false},
    }
    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {   // sub-test: names and isolates each row
            if got := IsValid(tc.raw); got != tc.want {
                t.Errorf("IsValid(%q) = %v, want %v", tc.raw, got, tc.want)
            }
        })
    }
}

And Rust, which has no built-in parameterization but reaches the same place with a helper (or the rstest crate). The repo’s kvlite protocol tests are a natural fit — one line goes to the server, one reply comes back:

#[test]
fn protocol_replies() {
    // (command sent, expected one-line reply)
    let cases = [
        ("PING",                 "PONG"),
        ("GET missing",          "NIL"),
        ("SET name ada lovelace", "OK"),   // value with spaces survives the wire
        ("DEL name",             "NOT_FOUND"),
    ];
    for (sent, want) in cases {
        let got = round_trip(sent);
        assert_eq!(got, want, "command {sent:?} should reply {want:?}");
    }
}

The shape is identical across all three: a table of rows, a single body that applies the same logic to each. Adding the next boundary case is adding one line to the table — not copying a function and hoping you edited every literal correctly.

Each row must fail independently and name itself

Here is the rule that separates a good table from a trap, and it is the whole reason the pattern earns its place: a failing row must tell you which row failed, and one failing row must not stop the others from running.

Consider the wrong way to write the loop:

# ANTI-PATTERN: a bare loop inside one test
def test_all_usernames():
    for raw, expected in CASES:
        assert is_valid(raw) is expected   # first failure aborts the whole test

This runs as one test. The moment any row fails, the assert raises and the loop stops — you learn that something is broken, but not which cases, and never whether the rows after the first failure would have passed. The failure report says test_all_usernames FAILED and nothing more. You have coupled independent cases into one fate.

The correct forms make each row a first-class test with its own identity:

• pytest.mark.parametrize generates a separate test node per row: test_is_valid[empty], test_is_valid[too_long]. Each passes or fails on its own; the report names the case.
• Go’s t.Run(tc.name, …) creates a named sub-test per row, isolated so a failure (or even a panic, contained) does not abort its siblings.
• In a hand-rolled Rust or JS loop, at minimum attach the case to the assertion message (assert_eq!(got, want, "command {sent:?} …")) so the failure output identifies the row — and prefer a framework mechanism that keeps rows independent when one exists.

   bare loop                         parameterized / sub-tests
   ─────────                         ─────────────────────────
   test_all  FAILED                  is_valid[too_short]  ok
     (which row? unknown)            is_valid[minimum_ok] ok
     (rows after it? never ran)      is_valid[maximum_ok] FAIL  ← the exact case
                                     is_valid[too_long]   ok
                                     is_valid[empty]      FAIL  ← and this one too

The second report is a diagnosis. It tells you maximum_ok and empty broke, that the others are fine, and — because both ran — that the fix has two boundaries to satisfy, not one. That is the difference between a test that finds bugs and a test that merely announces their existence.

A corollary: name your rows. test_is_valid[case3] is barely better than a bare loop; test_is_valid[boundary_max_length] tells the next person what the row is for and makes a missing row conspicuous. The names are documentation of your equivalence classes.

Generating cases from data or generators

Sometimes the table itself comes from somewhere else — a CSV of real examples, a JSON fixture, a golden file, or a generator. This is the “data-driven” end of the spectrum, and it is powerful and double-edged.

External data files

Loading cases from a file (cases.csv, a fixtures directory, a set of input → expected golden files) shines when the cases are numerous, when non-programmers curate them, or when they are captured from production. A localization team can add a row to a spreadsheet without touching test code; a batch of real malformed inputs from an incident can become a regression corpus verbatim.

The cost is intent. A literal table in the test file says, at a glance, why each row exists — this is the max-length boundary, this is the empty case. A thousand rows in a CSV say only that they exist. When the data lives elsewhere, a reader can no longer see the equivalence classes you meant to cover, and a reviewer cannot tell a deliberate boundary from a random sample. Use external data when the volume or ownership genuinely demands it; keep the boundary cases that encode your design in the code, where they document intent.

Property-based generators

A property-based test inverts the idea: instead of you listing rows, a generator produces hundreds of random inputs and checks that a property holds for all of them — “for any list, sorting it twice gives the same result as sorting it once,” “for any valid username, parse(render(u)) == u.” Tools like Hypothesis (Python), QuickCheck (Haskell), proptest (Rust), and fast-check (JS) also shrink a failing input down to the minimal example that still breaks — turning a 10,000-element counterexample into the two-element one you can actually reason about.

example-based                     property-based
─────────────                     ──────────────
you enumerate rows                generator enumerates for you
covers the cases you thought of   explores cases you didn't
a row = a claim about one input   a property = a claim about ALL inputs
misses the input you forgot       finds the input you forgot, then shrinks it

Property-based testing is the natural ally of the edge-cases Part: it is very good at discovering the empty collection, the duplicate, the Unicode surprise, the integer at the boundary — precisely the inputs a human enumeration forgets. When it helps: invariants and round-trips over a large input space. When it obscures: when the “property” degenerates into re-implementing the function under test inside the test (now you have two implementations of the same bug), or when a random failure is hard to reproduce because the seed was not logged. Pin the seed, keep properties genuinely independent of the implementation, and treat a shrunk counterexample as a new row for your example-based table.

Connecting to test design

Step back and the reason this Part cares about parameterization becomes clear. In tester-mindset and edge-cases you learned to enumerate cases: the boundaries of a range, the members of each equivalence class, the zero and the MAX and the empty. That enumeration is only valuable if running it is cheap. Parameterization is what makes it cheap.

   test DESIGN says WHICH cases        parameterization makes them CHEAP
   ────────────────────────────        ─────────────────────────────────
   boundary-value analysis   ─────►    one row per boundary
   equivalence partitioning  ─────►    one row per class representative
   the enumerated sad paths  ─────►    one row per failure mode

The table is your test design, written down. Each row is a boundary or a class representative; the set of rows is a visible, reviewable claim about what you cover. A reviewer scanning the table can see that the empty case is present and the max-length case is present — or notice that they are absent. Coverage design becomes something you can read, not something buried across twenty look-alike functions.

By the numbers — the case-count explosion

A function of 3 inputs, each with 4 boundary values you must cover:

  as copy-pasted functions:  up to 4 × 4 × 4 = 64 near-identical test bodies
  as a parameterized table:   1 test body + 64 rows (often far fewer, chosen
                              by pairwise/combinatorial selection)

Editing the assertion style:
  copy-pasted:   64 edits, any of which you can miss
  parameterized:  1 edit, applied to every row at once

The table also makes a gap visible: 64 expected rows, 61 present → three
boundaries are silently untested, and you can SEE the three missing lines.

The architect’s lens

• Why does it exist? Because the real structure of most tests is “same logic, different data,” and writing that as N copied functions duplicates the logic N times while scattering the data — parameterization factors the shared body out and lines the data up as a table.
• What problem does it solve? It removes the maintenance liability of copy-pasted tests and, more importantly, makes coverage gaps visible: a missing boundary is a missing row you can see, not a missing function hidden among look-alikes. It is what makes boundary-value and equivalence-class testing cheap enough to actually run.
• What are the trade-offs? A table trades some readability of a single case for readability of the set; a badly built one (bare loop, unnamed rows) hides which case broke and lets one failure mask the rest. Data-driven and property-based variants add power but can obscure why each case exists.
• When should I avoid it? When cases share almost no logic (forcing unrelated tests into one contorted body), or when a single high-value scenario reads better as its own well-named test. Parameterize repetition, not everything.
• What breaks if I remove it? Boundary and equivalence testing get expensive, so people skip cases; the suite fills with copy-paste drift; and a failure tells you “something is wrong” instead of “these three specific boundaries broke.” You lose both cheap coverage and precise diagnosis.

Check your understanding

1. Beyond being repetitive to type, name the two concrete risks that copy-pasted near-identical tests introduce, and explain why a missing case is easier to spot in a table than in a wall of separate functions.
2. What is a parameterized (table-driven) test, and what does a single row typically contain?
3. Why is a bare for loop with an assert inside one test function an anti-pattern? What do pytest.mark.parametrize and Go’s t.Run do that the bare loop does not?
4. When does loading test cases from an external CSV or fixtures directory help, and what does it cost compared with a literal table in the test file?
5. How does parameterization connect to boundary-value analysis and equivalence partitioning — why is the table itself a form of test design, and where do property-based generators fit?

Show answers

1. (a) A maintenance liability: a change to the call or assertion shape must be made in every copy, and a missed one leaves the suite quietly inconsistent; (b) hidden coverage gaps and drift: because each case is its own hand-written function, the absence of a case is invisible, and copy-paste error can leave an input edited but its expectation not. In a table a missing case is a missing row you can see; among many similar functions, a missing function is invisible.
2. One test body executed once per row of a table of cases, where each row is typically a tuple of (input…, expected) — often with a name — separating the data that varies from the logic that stays the same.
3. A bare loop runs as one test: the first failing assert raises and aborts the loop, so you learn only that something broke, not which rows, and the rows after the first failure never run. pytest.mark.parametrize generates a separate, named test node per row and t.Run creates an isolated named sub-test — so every row passes or fails independently and the report names the exact failing case(s).
4. External data helps when cases are very numerous, are curated by non-programmers, or are captured from production (e.g. a regression corpus from an incident). It costs intent: a literal table shows at a glance why each row exists (this is the max boundary, this is the empty case), whereas a thousand external rows show only that they exist, hiding the equivalence classes and boundaries from readers and reviewers.
5. Test design (boundary-value analysis, equivalence partitioning, enumerated sad paths) decides which cases matter; parameterization makes running them cheap, so the table becomes your test design written down and reviewable — one row per boundary or class representative, with missing rows visible. Property-based generators sit at the other end: instead of you listing rows, they generate many inputs and check a property holds for all, discovering the edge cases you forgot and shrinking a failure to a minimal example you can turn into a new row.