Skip to content
Software Testing · First Principles

Managing Test Data

The previous page gave your tests a stable place to run. This page is about what goes inside that place: the data. A test almost never runs against an empty system. It needs a user to log in as, an order to refund, a document to search for. Where that data comes from — and who owns it — is one of the quietest determinants of whether a suite is trustworthy or a slow, brittle liability.

The core tension of this page is a single trade-off, and everything else is a strategy for navigating it. Realistic data finds real bugs; minimal data keeps tests fast and their intent obvious. A test seeded from a scrubbed copy of production will surface the edge case nobody imagined — the customer with an emoji in their name, the order with 4,000 line items. A test with exactly one hand-crafted user makes it blindingly clear what is being verified. You want both, and you cannot have both in the same test. Learning which to reach for, when, is the skill.

The strategies, from static to dynamic

Section titled “The strategies, from static to dynamic”

There are three broad ways to get data in front of a test, and they sit on a spectrum from fixed and cheap to flexible and expensive.

   static fixtures        factories / builders        database seeding
   ────────────────       ────────────────────        ────────────────
   a JSON/SQL file        code that manufactures       load a whole dataset
   loaded verbatim        an object on demand          into a real DB
   fast, but frozen       expressive, overridable      realistic, but heavy
   unit-ish tests         unit + service tests         integration / e2e

Static fixtures — fast, and brittle

Section titled “Static fixtures — fast, and brittle”

A fixture is a fixed blob of data — a user.json, a seed.sql, a hard-coded dictionary — that a test loads verbatim. Fixtures are fast (no computation, just a read) and simple. That is their whole appeal.

Their weakness is that they are frozen, and your schema is not. The day someone adds a required country_code column, every fixture that predates it is silently wrong. If validation is loose, the stale fixture loads anyway and your test passes against data that could never exist in production — false confidence. If validation is strict, forty unrelated tests go red at once — false failures. Both outcomes trace to the same root: a fixture is a copy of the schema’s shape, made at one moment, that nobody updated when the shape changed. This is a leading cause of mysterious CI breakage.

The second weakness is drift by accretion. A shared users.json starts with two users. Someone needs an admin, so they add a third. Someone needs a locked-out account, a fourth. A year later it has thirty users and no test dares delete one, because who knows what depends on it?

Factories and builders — flexible, and expressive

Section titled “Factories and builders — flexible, and expressive”

A factory (or builder) is not data; it is code that makes data on demand. Instead of a frozen user.json, you write a function that returns a valid user, with sensible defaults for every field, and lets each test override only the fields it cares about.

# A factory: valid defaults, overridable per test.
def make_user(**overrides):
    base = {
        "id": next_id(),
        "email": f"user{next_id()}@example.test",
        "country_code": "CA",
        "status": "active",
        "created_at": clock.now(),   # injectable clock, per the next page
    }
    return User(**{**base, **overrides})


# A test states ONLY what matters to it. Everything else is a sane default.
def test_locked_user_cannot_log_in():
    user = make_user(status="locked")     # the one relevant field
    assert login(user, "any-password") is Denied

Look at what the override bought you: the test reads as a sentence. “A locked user cannot log in.” No country_code, no created_at, no noise — those exist because the object must be valid, but the factory keeps them out of the test’s face. This is the single most important property of good test data: the data a test sets up should be exactly the data the test is about, and nothing else. A reader should be able to see why each value is there.

Factories also solve the schema-drift problem structurally. When country_code becomes required, you add it once, in the factory’s defaults, and every test keeps working. The factory is a single place where “what does a valid object look like today?” lives — and it lives in code that runs, so it cannot silently rot the way a JSON file can.

Database seeding — realistic, and heavy

Section titled “Database seeding — realistic, and heavy”

Unit-level factories build objects in memory. Integration and end-to-end tests need data that actually exists in a real database, because that is the whole point — you are testing the query, the constraint, the join. Seeding is loading a known dataset into a real datastore before the test runs.

The kvlite integration tests in this repo show the disciplined version. Each test does not share one database; it creates its own, on a path unique to that test:

// rust/kvlite/tests/server.rs — every test owns a fresh, isolated store.
fn temp_wal() -> std::path::PathBuf {
    static N: AtomicU64 = AtomicU64::new(0);
    std::env::temp_dir().join(format!(
        "kvlite-srv-{}-{}.wal",
        std::process::id(),                       // unique to this process
        N.fetch_add(1, Ordering::Relaxed)         // unique to this test
    ))
}


#[test]
fn tcp_set_get_del_roundtrip() {
    let path = temp_wal();
    let _ = std::fs::remove_file(&path);          // start from nothing
    let (db, _) = Db::open(&path).unwrap();
    // ...test drives THIS db, which no other test can see...
}

That temp_wal() is a factory for databases. Because the path embeds the process id and an atomic counter, two tests — even running in parallel — can never touch the same file. There is no shared golden database to corrupt, and no ordering dependency between tests. That property has a name, and it is the heart of this whole page.

Every test owns and isolates its own data

Section titled “Every test owns and isolates its own data”

The governing rule is short: a test sets up the data it needs, and cleans up after itself, and never depends on data another test left behind.

The reasoning is first-principles. A test suite is only trustworthy if each test’s result depends only on the code under test — not on which other tests ran, or in what order, or whether one of them crashed halfway. The moment test B relies on a row that test A happened to insert, you have coupled them: reorder the suite, run B alone, run them in parallel, and B fails for a reason that has nothing to do with its assertion. You have manufactured a flaky test out of thin air — which is exactly the poison the next page is about.

Isolation has three common shapes, cheapest last:

   transaction rollback   truncate + reseed        fresh database
   ────────────────────   ─────────────────        ──────────────
   BEGIN; ...test...;      delete all rows,         spin up a new DB
   ROLLBACK;              re-seed known data        (or temp file)
   fast, per-test         medium                    slow, bulletproof
   needs 1 connection     ordering-sensitive        e.g. temp_wal()

Wrapping each test in a transaction that is rolled back at the end is the fastest and most popular pattern: the database is pristine again the instant the test finishes, at almost no cost. Truncate-and-reseed is the fallback when the code under test manages its own transactions. A brand-new database per test — like temp_wal() — is the slowest but the most bulletproof, and it is the right call for anything touching persistence, replication, or migrations.

The mystery guest antipattern

Section titled “The mystery guest antipattern”

There is one specific failure of ownership worth naming, because it is so common and so corrosive. A test reads like this:

def test_premium_user_sees_dashboard():
    resp = client.get("/dashboard", user_id=42)   # ...who is user 42?
    assert resp.status == 200
    assert "Premium" in resp.body

Why 42? Why premium? The answer lives somewhere else entirely — in a shared seed.sql loaded before the suite, in which user 42 happens to have a premium plan. This is the mystery guest antipattern: the test silently depends on data it does not show you, so you cannot understand it by reading it. You have to go find the fixture, cross-reference the id, and reconstruct the setup in your head.

The mystery guest is doubly bad. It makes the test unreadable — the intent is offstage — and fragile, because anyone who edits the shared seed can break user 42’s premium status and send this test red for a reason invisible from its own body. The cure is the factory pattern above: build the premium user right there, in the test, so the data and the assertion sit side by side.

def test_premium_user_sees_dashboard():
    user = make_user(plan="premium")               # the guest is named, here
    resp = client.get("/dashboard", user_id=user.id)
    assert "Premium" in resp.body

Prefer explicit, local data. If a reader must scroll to another file to know what a test is about, the test has a mystery guest.

Anonymized production data — real shapes, hard privacy

Section titled “Anonymized production data — real shapes, hard privacy”

Hand-built data has a blind spot: you only build the cases you already thought of. Production data is the opposite — it is a catalogue of every shape real users actually produce, including the ones no engineer would invent. Feeding a scrubbed copy of production through your tests is the surest way to find the bug that only triggers on the 200-character name, the negative balance, the order placed during a leap second.

But production data is made of people, and that turns it into a hard constraint, not a nice-to-have. A copy of the users table is names, emails, addresses, payment tokens — personally identifiable information (PII). The instant it leaves the production boundary and lands in a test database, a laptop, or a CI log, you have created a privacy liability and, in most jurisdictions, a legal one. Treat this as an absolute: test data must never contain real PII. There are two disciplined ways to honor that rule.

  • Anonymization / scrubbing — take real production data and irreversibly replace every sensitive field: names become fake names, emails become user-<hash>@example.test, card numbers become test tokens. The shapes survive (string lengths, distributions, null patterns, weird unicode) but no real person does. The word “irreversibly” is load-bearing — pseudonymization that can be reversed with a lookup table is not scrubbing, it is a breach waiting to be joined back together.
  • Synthetic generation — manufacture data that statistically resembles production (same distributions, same edge-case frequencies) without ever touching a real record. Nothing to scrub, because nothing real went in.

The architect’s lens

Section titled “The architect’s lens”
  • Why does it exist? Because tests need data to run against, and hand-managing that data naively — one giant shared fixture — produces suites that are slow, unreadable, and coupled. Deliberate test-data strategy exists to keep data owned, isolated, and honest about intent.
  • What problem does it solve? It navigates the central trade-off between realistic data (finds real bugs) and minimal data (fast, focused, obvious), and it stops one test’s data from silently breaking another’s.
  • What are the trade-offs? Static fixtures are fast but drift and rot against the schema; factories cost a little authoring effort but stay in sync and read clearly; seeded/production data finds real edge cases but is slow and carries privacy risk. You pay in speed, effort, or risk — pick knowingly.
  • When should I avoid it? Avoid a big shared seed database when a per-test factory would do; avoid real production data anywhere PII cannot be provably scrubbed; avoid realistic-data heaviness in the fast unit tests that should stay obvious.
  • What breaks if I remove it? Drop isolation and tests become order-dependent and flaky. Drop factories and you get mystery guests and stale fixtures. Drop scrubbing and you turn a test database into a data breach.

Check your understanding

Section titled “Check your understanding”
  1. A colleague says “just add the user you need to seed.sql and reference its id in the test.” What antipattern is that, and what are its two distinct harms?
  2. Contrast static fixtures and factories specifically on what happens when a new required column is added to the schema. Why does one rot and the other not?
  3. Why does the kvlite temp_wal() helper embed both the process id and an atomic counter in the filename? What property would be lost if it used a single fixed path?
  4. Your team wants to catch bugs that only appear on real-world data, so they propose copying the production database into the test environment. State the one hard requirement and one concrete way the scrub can still leak PII.
  5. Restate the central trade-off of this page in one sentence, and explain why a healthy suite ends up using both ends of it.
Show answers
  1. The mystery guest antipattern: the test depends on data it does not show you. The two harms are that it is unreadable (you must go find the fixture to know what “user 42” even is) and fragile (anyone editing the shared seed can break the test for a reason invisible from its own body). The fix is to build the user locally with a factory.
  2. A static fixture is a frozen copy of the schema’s shape; when a required column is added, the fixture is silently wrong — either it loads anyway and gives false confidence, or it fails validation and produces false failures across many tests. A factory keeps “what a valid object looks like today” in one place in code, so you add the column’s default once and every test keeps working. Data-as-code can be updated centrally; data-as-frozen-file cannot.
  3. So that no two tests — even running in parallel — ever touch the same file: the process id separates test runs, the atomic counter separates tests within a run. With a single fixed path you lose isolation: tests would share state, become order-dependent, and one test’s leftover data (or crash mid-run) could fail an unrelated test — manufactured flakiness.
  4. The hard requirement: the data must never contain real PII — every sensitive field must be irreversibly scrubbed (or the data synthesized). It can still leak because PII hides in forgotten columns — a free-text notes field, a metadata JSON blob, a support-ticket body — which a denylist scrub never looks at. Scrub with an allowlist (redact by default), and give the copy production-grade access controls.
  5. Realistic data finds real bugs; minimal data keeps tests fast and their intent obvious — and you cannot maximize both in one test. A healthy suite uses minimal, factory-built data for the fast majority of tests (clear intent, quick feedback) and a small number of realistic-data tests to guard the edge cases the factories would never think to construct.