Test Environments & Configuration
The part overview argued that the real world is where clean test theory meets messy operational reality. Nowhere is that clash sharper than the environment a test runs in. A test does not run in the abstract. It runs on a machine, against some dependencies, with some configuration, holding some secrets. Change any of those and the same test code can pass here and fail there — or worse, pass everywhere and still let a defect reach production.
This page is about earning justified confidence despite that. The throughline of the book asks how you know software works. Part of the honest answer is: it works in an environment. If your tests run in an environment that does not resemble production, a green suite is not evidence — it is a comforting story. “Works on my machine” is not a defense you offer when something breaks. It is the name of the bug.
”Works on my machine” is a bug report
Section titled “”Works on my machine” is a bug report”When an engineer says “works on my machine,” they have, without meaning to, filed a precise bug report. They have proven the software works in one specific environment and failed to reproduce that environment anywhere else. The defect is not in the code the test exercised. The defect is in everything the test assumed and never stated: a library version, a locale, a timezone, an env var that happened to be set, a database that happened to already have a row in it.
The whole discipline of test environments is the practice of making those assumptions explicit and reproducible, so that “it works” means “it works anywhere this environment is recreated,” not “it works where I happened to be standing.”
Hidden assumption Made explicit ───────────────── ───────────── "Node 18 is installed" ──► pinned in CI image / .nvmrc "TZ is America/Toronto" ──► TZ=UTC set for the test run "the DB already has a user" ──► test seeds its own user "PAYMENTS_URL points at stub" ──► injected config, asserted at start "my ~/.aws creds work" ──► scoped non-prod creds, injectedThe environment ladder
Section titled “The environment ladder”Tests do not all run in the same place, and they shouldn’t. There is a ladder of environments, each closer to production and each more expensive to run against. Climbing it trades speed and isolation for realism.
┌──────────────────────────────────────────────────────────┐ │ production real users, real money, real data │ ← never test here ├──────────────────────────────────────────────────────────┤ │ production-like / a full deployed stack, prod config │ ← e2e, smoke, │ staging shape, synthetic data │ release checks ├──────────────────────────────────────────────────────────┤ │ CI clean, ephemeral, reproducible; │ ← unit + most │ containers for real dependencies │ integration ├──────────────────────────────────────────────────────────┤ │ local the engineer's laptop; fast loop │ ← unit, quick │ │ integration └──────────────────────────────────────────────────────────┘ cheaper / faster / more isolated ▲ slower / more realistic / shared ▼- Local — the engineer’s machine. The fast inner loop. Optimized for speed and immediate feedback, not fidelity. Whatever passes here must still be proven in CI, because local is the least reproducible rung.
- CI — a clean, disposable machine that runs the suite on every push. Its superpower is that it starts from a known image every time. If something passes in CI, it passes on a machine nobody has customized.
- Staging / production-like — a deployed copy of the whole system, wired to prod-shaped dependencies with synthetic data. This is where end-to-end and smoke tests earn their keep, because it is the first rung where the integration of everything is real.
- Production — you do not run destructive tests here. You observe here (health checks, synthetic monitors, canaries), which is a different discipline covered elsewhere.
The ladder is not a sequence you pass through once. It is a set of gates a change crosses on its way to users: fast checks locally to keep the loop tight, the full suite in CI on every push to catch what your machine hid, and the deployed layers before release to catch what only shows up when the whole system is assembled. A change that only ever passed on rung one has not been tested — it has been sampled.
The parity gap — why a test lies
Section titled “The parity gap — why a test lies”The distance between where a test runs and production is the parity gap. Every difference across that gap is a place where a passing test can be wrong about production.
test environment production ──────────────── ────────── SQLite in memory ≠ Postgres 15 cluster single process ≠ 12 replicas behind a load balancer TZ unset (→ UTC) ≠ TZ=America/Toronto 100 rows ≠ 40 million rows stubbed payment gateway ≠ real gateway with rate limits └── each ≠ is a lie the test can tell ──┘A test that passes against SQLite but runs against Postgres in prod has not tested your production query. It has tested a different query that happened to look the same. Closing the parity gap does not mean making every rung identical to prod — that would be unaffordable and slow. It means being honest about which gaps remain and covering the risky ones higher up the ladder, where parity is real.
The practical rule: for each gap, ask “could this difference change behavior?” A different log format cannot cause a defect, so leave that gap open. A different database engine, a different timezone, a different concurrency model — those can, so they must be closed by testing on a rung where the real thing is present. You are not chasing perfect parity; you are spending your parity budget on the differences that can lie.
Configuration is an explicit input
Section titled “Configuration is an explicit input”A test has inputs — the arguments you pass, the fixtures you load. Configuration is also an input, and the failure mode is treating it as invisible ambient state instead. The twelve-factor app principle states the rule plainly: strict separation of config from code, with config supplied by the environment, not baked into the artifact. The same built image runs in CI, staging, and prod; only the injected config differs.
For tests this has a hard consequence: never hardcode an environment assumption into a test. A test that contains https://api.staging.internal is not a test of your code; it is a test welded to one environment that will silently rot when that host moves.
config source typical use in tests ───────────── ─────────── ──────── environment vars DB_URL, LOG_LEVEL, TZ set per test run; the (the 12-factor default) most portable knob
config files structured, versioned a test-only profile, defaults per environment overridden by env vars
feature flags toggle behavior at runtime set the flag explicitly; without redeploying assert against a known stateMake config visible at the start of a test run. Log the resolved DB_URL, the active feature-flag set, the timezone. When a test fails in CI but passes locally, the first question is always “what was different?”, and the answer is almost always config. A run that prints its own configuration answers that question in seconds instead of an afternoon.
Feature flags deserve a special warning here, because they multiply the number of environments you actually have. Ten independent boolean flags is not one system — it is up to 1,024 possible configurations, and only a handful are ever tested. When a test asserts behavior, it must set the flags it depends on explicitly rather than inheriting whatever default happens to be live. A test that silently relies on a flag being off will pass today and mysteriously break the morning that flag is flipped on in staging — and by then the test’s green history has trained everyone to trust it.
Secrets in test environments
Section titled “Secrets in test environments”Real systems need credentials: database passwords, API keys, signing keys. Tests need them too, and this is where environment discipline meets security. Three rules cover almost everything.
- Injected, never committed. A secret in the git history is compromised the moment it lands, even if later deleted — clones and forks keep it forever. Secrets arrive at runtime through environment variables or a secrets manager, and the repo holds only a reference (a variable name), never a value. The
.cloudflare.env.exampleconvention in this repo is exactly this: a committed template of names with placeholder values, and the real values live only in the untracked.env. - Scoped to non-prod. A credential used in a test environment must not be able to touch production. Use a separate, least-privileged account whose blast radius is a throwaway database or a sandbox tenant. If a test leaks a key or a rogue test deletes everything, the damage is confined to a disposable world.
- Rotatable and observable. Assume any test secret will eventually leak. Prefer short-lived, rotatable credentials, and make it possible to see when one was last used, so a leak is detectable rather than silent.
The mental test is simple: if this exact secret appeared in a public gist tomorrow, what could an attacker do? If the answer is “reach production,” the setup is wrong regardless of how careful everyone is.
There is a second, subtler failure to avoid: a test that depends on a real secret is a test that cannot run for someone without it — a new contributor, a CI job on a fork, an offline machine. Push credential-requiring tests up the ladder (integration and above), keep the unit layer credential-free, and gate the higher layers so they skip cleanly when the secret is absent rather than failing with a confusing auth error. A test suite that a stranger can clone and run — minus the layers that legitimately need secrets — is a suite whose green result other people can trust.
Match the test level to the environment
Section titled “Match the test level to the environment”The test pyramid is not only about how many tests. It is about where each layer runs. Each level needs — and deserves — a different environment.
level needs runs on ───── ───── ─────── unit nothing external; pure functions any machine, no setup integration real dependencies (DB, queue) containers / ephemeral DB e2e the whole deployed stack staging / production-like- Unit tests need no environment. They exercise a function with in-memory inputs. If a “unit” test needs a database, it is an integration test wearing a disguise, and it will be slow and flaky in proportion to that lie.
- Integration tests need real dependencies. Not a mock of Postgres — Postgres. The whole point is to catch the parity gaps a mock hides: the query the mock accepted but the real engine rejects, the constraint the mock did not enforce. This is what containers and ephemeral databases are for.
- End-to-end tests need a deployed stack. They drive the system the way a user does, across process and network boundaries, so they need everything actually running — hence staging.
This repo’s own tests show the split. Its unit tests exercise pure logic with no I/O. Its integration test in rust/kvlite/tests/server.rs stands up a real TCP server and drives it with a real client socket — the file’s own comment says it does “exactly what nc would do by hand” — because a mock socket would not prove the wire protocol works.
The wrong instinct is to run everything at the highest level “to be safe.” An e2e test that could have been an integration test is slower to run, flakier when it fails, and vaguer about what broke — a failing end-to-end assertion tells you the system is wrong somewhere across a dozen components. Push each check to the lowest rung that can still catch its class of bug. That is the same economy the pyramid encodes: many cheap, precise tests at the bottom; a few expensive, realistic tests at the top; and the environment for each chosen to match, not exceed, what the level actually needs.
Ephemeral beats shared
Section titled “Ephemeral beats shared”Here is the single highest-leverage decision on this page. Given a choice between a shared, long-lived test environment and an ephemeral, disposable one, choose ephemeral.
A shared “test database” or a single “staging” that every engineer points at feels efficient — set it up once, everyone uses it. In practice it drifts (its schema and data diverge from prod through months of manual pokes) and it collides (two engineers’ tests fight over the same rows, and a failure could be your bug or theirs). You can never fully trust a result, because you can never fully control the starting state.
An ephemeral environment is created for a run and destroyed after: spin up, test, tear down. Its starting state is known because you just built it. There is no collision because it is yours alone. There is no drift because it never lives long enough to drift.
Shared, long-lived Ephemeral, disposable ────────────────── ───────────────────── set up once, reused forever created per run, destroyed after drifts from prod over months fresh from a known definition engineers collide on shared state isolated — one owner, one run "is this my failure or theirs?" failures are unambiguous green run ≠ prod-safe green run is real evidenceThe kvlite tests embody this at the smallest scale. Each test binds to 127.0.0.1:0 — port zero means “let the OS hand me any free port,” so parallel tests never collide on a fixed port. Each uses a temp_wal() file namespaced by process id and an atomic counter, so no two tests share a data file. Spin up, test, tear down — no shared state to collide on, nothing left behind to drift.
That pattern scales all the way up. Swap the temp file for a throwaway Postgres container and the fixed port for an OS-assigned one, and a full integration environment inherits the same three properties: known starting state, single owner, no persistence to drift. The mechanisms differ by scale; the principle — own your world, then destroy it — is identical from a temp file to a preview stack.
Under the hood — how ephemeral is built
Section titled “Under the hood — how ephemeral is built”Ephemerality is a technique with a few standard mechanisms, from smallest to largest scope:
- Port 0 / random ports. As above: never bind a fixed port in a test, or two tests (or two engineers) collide. Ask the OS for a free one.
- Temp files and temp directories. Namespace by process id plus a counter (as kvlite does) so concurrent tests never share a path. Clean up on teardown; a fresh temp dir per run makes cleanup a non-issue.
- Containers for dependencies. Libraries in the Testcontainers family (available across Java, Go, Python, Node, Rust, and more, as of mid-2026) start a real Postgres/Redis/Kafka in a throwaway container, hand your test a connection string, and destroy it after. Real dependency, zero shared state.
- Ephemeral preview environments. At the top of the ladder, a whole stack spun up per pull request from infrastructure-as-code, then torn down on merge. Expensive, but it gives each change its own production-shaped world.
The three failure modes, named
Section titled “The three failure modes, named”Almost every environment-related test failure is one of these three. Naming them makes them diagnosable.
- Environment drift. A long-lived environment slowly diverges from prod — an unpinned dependency updates, someone hand-edits staging’s schema, a config is changed and never written down. Tests keep passing against a world that no longer resembles production. The fix: pin versions, rebuild environments from definitions, prefer ephemeral over long-lived.
- Shared-state collisions. Two runs touch the same mutable state — the same DB row, the same fixed port, the same account — and interfere. Symptoms are the worst kind: flaky, order-dependent, “passes when I run it alone.” The fix: isolation. Give every run its own data, its own port, its own tenant.
- Config differences. The test environment’s config differs from prod in a way nobody noticed — a flag on in staging but off in prod, a different timeout, a different DB engine. The suite is green and the deploy still causes an incident. The fix: treat config as an explicit input, make it visible, and cover the risky differences with tests high on the ladder.
Notice how tightly these braid together with the rest of this part. Shared-state collisions are a leading cause of the flakiness you will chase in Determinism and diagnose in Debugging a Failing Test. The data those environments hold is the subject of Managing Test Data. And an environment that has quietly drifted is exactly the kind of test debt tracked in Test Maintenance & Test Debt. Environment discipline is upstream of most of it.
The architect’s lens
Section titled “The architect’s lens”- Why does it exist? Because software runs in an environment, and a test only proves what it proves in the environment it ran in. Deliberate test environments exist so that “it works” means “it works anywhere this environment is recreated,” not “it works where I stood.”
- What problem does it solve? The parity gap — the distance between where tests run and production — which lets a green suite coexist with a broken deploy. Environment discipline shrinks the gap where the risk is, and makes the remaining gaps visible.
- What are the trade-offs? Realism costs speed and money. Climbing the ladder toward prod parity catches more, but each rung is slower and more expensive, so you cannot run everything at the top. You buy fidelity with time.
- When should I avoid it? Do not pay for high parity where you do not need it. A pure unit test needs no environment; forcing a database or a container onto it only makes it slow and flaky. Match the level to the environment, and no higher.
- What breaks if I remove it? Tests become non-reproducible. “Works on my machine” becomes the norm, shared environments drift and collide, secrets leak into repos, and a green CI run stops being evidence — it becomes a story you tell yourself right up until the prod incident.
Check your understanding
Section titled “Check your understanding”- Why is “works on my machine” better understood as a bug report than as a defense?
- Name the four rungs of the environment ladder from cheapest to most realistic, and say what each one is best at testing.
- What is the “parity gap,” and how can a test pass while still being wrong about production?
- State the twelve-factor rule for configuration and give one way you would apply it so a test is not welded to a single environment.
- Give the three rules for handling secrets in test environments, and explain why ephemeral environments beat shared long-lived ones.
Show answers
- It is a claim that the software works in exactly one unreproducible environment, which means the real defect is in the unstated assumptions (versions, locale, timezone, ambient config, pre-existing data) that were never made explicit or recreated. Making those assumptions explicit and reproducible is the whole discipline.
- Local (fast inner loop, unit and quick integration), CI (clean and reproducible, unit plus most integration against containerized real dependencies), staging / production-like (a deployed stack, e2e and smoke tests), production (never destructively tested — observed via health checks, synthetic monitors, canaries).
- The parity gap is the distance between where a test runs and production. A test can pass against a different-but-similar-looking setup (SQLite instead of Postgres, UTC instead of the prod timezone, 100 rows instead of millions, a stubbed gateway instead of the real one); it exercised something that resembles prod, so its green result is not evidence about the real thing.
- Twelve-factor: strict separation of config from code, with config injected by the environment so the same built artifact runs everywhere and only config differs. Apply it by reading values like
DB_URLfrom environment variables rather than hardcodinghttps://api.staging.internalinto the test, so the same test runs unchanged in any environment. - Secrets must be (a) injected at runtime and never committed, (b) scoped to non-prod least-privileged accounts so a leak cannot touch production, and (c) rotatable and observable so leaks are detectable. Ephemeral environments (spin up, test, tear down) beat shared long-lived ones because they start from a known state, are owned by a single run so nothing collides, and never live long enough to drift from production.