Running in CI — Fast, Deterministic, and Green

The last few pages built the suite: what to automate, the pyramid that shapes it, the frameworks that run it, and page objects and data-driven tests that keep it maintainable. All of that produces one thing: a command you can type on your machine that says pass or fail.

But a suite that only runs when you remember to type the command protects nobody. Your teammate merges without running it. You’re on holiday. The one time it mattered, someone was in a hurry. Automation only pays off when it runs automatically — on every change, without a human deciding to. That is what CI is for, and it is the whole point of everything we’ve built so far.

Why CI is the point of automation

Recall the throughline of this book: how do you earn justified confidence that software works? A test result you have to remember to produce is not justified confidence — it’s hope with extra steps. The confidence has to be automatic, tied to the act of changing the code, or it decays the moment attention slips.

Continuous Integration is the machine that makes it automatic. Every push, every pull request, a neutral server checks out the change, builds it, and runs the suite — the same way, every time. No one has to remember. No one can claim “it works on my machine.” The build is either green (every gate passed) or red (something broke), and that signal arrives minutes after the code was written, while the author still has the context in their head.

This book is not the place to build that pipeline from scratch — the sibling DevOps · First Principles book does exactly that, in its Continuous Integration chapter and the surrounding pipelines and stages pages. What matters here is the tester’s half of the bargain: the suite you hand to that pipeline has to be fast, deterministic, and treated as a contract. A suite that is slow, flaky, or routinely ignored will get switched off no matter how good the pipeline is. Those three properties are what this page is about.

   developer pushes ──► CI server
                          │
                          ├─ checkout + build
                          ├─ lint / typecheck        (seconds)
                          ├─ unit tests   ─ tier 1 ─ (seconds)      fail fast
                          ├─ integration  ─ tier 2 ─ (a minute)         │
                          └─ end-to-end    ─ tier 3 ─ (minutes)         ▼
                                                              red ─► block merge
                                                              green ─► allow merge

The pipeline is a set of gates. A change may only merge if it passes every gate. That single rule — you cannot merge a red build — is what turns a pile of test files into a defence.

Keeping the suite fast

Speed is not a nice-to-have. It is the difference between a suite that gets run and one that gets skipped. There is a hard human threshold: if the feedback loop is longer than a coffee break, developers stop waiting for it. They push and context-switch, and the red result arrives after they’ve moved on — so the whole “catch it while it’s fresh” benefit evaporates. Slowness doesn’t just waste minutes; it silently disables the suite by making people route around it.

Four levers keep a suite fast.

Tiered gates — run the pyramid in layers

The pyramid is not just an authoring guide; it’s a scheduling guide. Run the cheap, broad base first and the expensive, narrow top last:

• Tier 1 — unit tests. Seconds. They run on every push and even in a pre-commit hook. If a unit test fails, stop here; there is no point spinning up a browser.
• Tier 2 — integration / service tests. Tens of seconds to a minute. Run on every pull request.
• Tier 3 — end-to-end / UI tests. Minutes. Run on the merge to the main branch, or nightly, or on a release candidate — not on every keystroke.

This is failing fast: order the gates cheapest-first so the most common failures are caught by the fastest checks, and you never pay for an expensive tier to tell you what a cheap one already knew.

Parallelism and sharding

Tests that don’t depend on each other can run at the same time. Parallelism runs multiple tests inside one machine (multiple cores); sharding splits the suite across multiple machines — shard 1 runs tests A–H, shard 2 runs I–P, and so on, each on its own runner. A 40-minute suite split across ten shards finishes in roughly four. Both only work if your tests are independent — which is the same discipline that makes them deterministic, below. This is not a coincidence.

By the numbers — the cost of a slow suite

Suppose a 100-person engineering org runs its full suite on every pull request, ~20 times a day per engineer including retries, and the suite takes 25 minutes.

wait per run            = 25 min
runs per engineer/day   = 20  (many are quick re-pushes)
engineers               = 100

lost engineer-minutes/day = 25 × 20 × 100 = 50,000 min ≈ 833 hours/day

Cut the suite to 5 minutes with sharding and tiered gates and you reclaim ~665 hours per day of waiting — before counting the context-switches each wait causes. Illustrative arithmetic, but the shape is real: suite latency is an org-wide tax paid continuously.

Determinism and flakiness

A deterministic test gives the same answer every time it runs on the same code: pass means pass, fail means fail. A flaky test sometimes passes and sometimes fails without the code changing. Flakiness is the single most corrosive disease a suite can catch, because it destroys the one thing tests exist to provide — a trustworthy signal.

Think about what a flaky test does to the green-build contract. A red build is supposed to mean “stop, something is broken.” But if builds go red at random, people learn to shrug and hit “re-run.” Once “re-run the flaky ones” is a habit, a real failure hides in the noise and sails straight through. One flaky test doesn’t cost you one test — it quietly poisons your trust in the entire suite.

The four causes of flakiness

Almost every flaky test traces to one of four root causes:

1. Shared state. Two tests read and write the same file, database row, global variable, or singleton. Test B passes alone but fails when Test A ran first and left state behind. The fix is isolation: each test sets up and tears down its own world (see the fresh-SharedStore-per-test pattern below).
2. Timing / races. The test sleeps for 200 ms and assumes the async work is done. On a loaded CI runner it takes 210 ms, and the test fails. The fix is to wait for a condition, not for a clock — poll until the expected state is reached, with a generous timeout.
3. Order dependence. The suite passes in the order you wrote it, but a test secretly relies on an earlier test having created a record. Run the suite shuffled and it collapses. The fix is the same as shared state: independence. Randomising test order in CI is a good way to surface this.
4. External dependencies. The test hits a real third-party API, a live clock, or the network, any of which can hiccup. The fix is to control them: test doubles for external services, and an injected/frozen clock instead of now().

Here is the isolation discipline in a real integration test — from the repo’s kvlite store. Every test constructs its own SharedStore, so no test can leave state behind for another:

#[test]
fn shared_store_from_many_threads() {
    // A brand-new, empty store. Nothing from any other test can reach it.
    let store: SharedStore<String, u64> = SharedStore::new();

    // Each thread writes its OWN key range, so every key is distinct and the
    // final length is exact — a deterministic assertion, not a "roughly N".
    let handles: Vec<_> = (0..THREADS)
        .map(|t| {
            let store = store.clone(); // same map, new handle
            thread::spawn(move || {
                for i in 0..PER_THREAD {
                    store.set(format!("t{t}-k{i}"), (t * PER_THREAD + i) as u64).unwrap();
                }
            })
        })
        .collect();
    for h in handles { h.join().unwrap(); }

    // Exact, not approximate. If a single update were lost, this fails every time.
    assert_eq!(store.len().unwrap(), THREADS * PER_THREAD);
}

Notice the two determinism moves: fresh state per test (no cross-test leakage) and exact assertions (assert_eq! on an exact count, not “at least most of them”). A test that asserts something approximate is a flaky test waiting to happen.

Zero tolerance for flaky tests

The only policy that works is zero tolerance. A test that flakes is broken, even if the code under it is fine — because a test’s job is to give a trustworthy signal, and a flaky one doesn’t. The moment a test is identified as flaky it must be either fixed immediately or quarantined: moved out of the blocking gate into a separate, non-blocking bucket, with a ticket to fix it and a deadline. Quarantine is not the same as ignoring. Ignoring means the test still runs, still goes red, and everyone has trained themselves to look away. Quarantine means it stops blocking the build but stays visible and owned, so the green signal for everyone else is honest again.

The trap to avoid at all costs is the automatic retry. Configuring CI to re-run failed tests up to three times “to reduce noise” feels pragmatic and is a slow catastrophe: it converts every flaky test into a silent one and lets genuinely intermittent bugs pass. Retries hide the disease instead of curing it.

It actually happened — Knight Capital, 2012

On 1 August 2012, the trading firm Knight Capital deployed new code to its servers. Reports indicate the deployment reached only some of the production machines, and an old, repurposed feature flag reactivated dormant code on the servers that were missed. In about 45 minutes the runaway system sent millions of unintended orders; the firm lost roughly $440 million and nearly went under.

The details are debated, but the testing lesson is not: this was a deployment and environment failure — inconsistent state across machines, dead code that no gate caught, no automated check that every server was running the same, expected version. A green build is only a contract if the thing that ships is exactly what the build tested, on every machine. Determinism doesn’t stop at the test runner; it runs all the way to production.

A green build as a contract

Once the suite is fast and deterministic, it can carry a promise: green means the change is safe to merge; red means stop. This is the software equivalent of the manufacturing andon cord — the line worker who can halt the entire assembly line when they see a defect. A red build is a stop-the-line event. The correct response is not “merge around it” or “someone will fix it later”; it is the build going red is now the team’s top priority until it’s green again.

This sounds strict, and it is, deliberately. The whole value of CI collapses the instant “red build” stops meaning “stop.” If the team routinely merges on red — because “that test is always flaky” or “my change is unrelated” — then green and red carry no information, and you’ve paid for a pipeline that certifies nothing. The contract only has value if it is never violated. That is why flakiness (which manufactures false reds) and merging-on-red (which ignores true reds) are two faces of the same failure: both sever the link between the signal and reality.

Quarantine, above, is how you keep the contract honest under pressure: instead of letting a flaky test turn the whole build red and tempt everyone to ignore it, you pull that one test out of the gate so the gate stays trustworthy — and you fix the pulled test on a clock, not “eventually.”

The recurring maintenance cost

A test suite is not a monument you build once. It is a living asset with a running cost, and pretending otherwise is how suites rot. Every test you write is code you must now maintain forever: when the product changes, tests must change with it; when a test breaks, someone must diagnose whether the test or the code is wrong.

Left untended, a suite decays in predictable ways: tests that assert obsolete behaviour, duplicate tests that all cover the same happy path, slow tests nobody dares delete, and flaky tests everyone has learned to ignore. Maintenance means actively pruning (deleting tests that no longer earn their keep) and refactoring (extracting shared setup, tightening slow tests, converting a fragile end-to-end test into a focused integration test). A suite of 10,000 tests where 2,000 are flaky or redundant is worse than a suite of 6,000 trustworthy ones — it costs more to run and trusts less.

Under the hood — why coverage numbers can lie

The most seductive maintenance metric is code coverage: the percentage of lines (or branches) executed by the suite. It is genuinely useful for finding code that no test touches. But it is a terrible proxy for confidence, and mistaking one for the other is a classic trap.

Coverage measures whether a line ran, not whether its behaviour was checked. Consider:

def test_discount():
    price = apply_discount(100, coupon="SAVE10")
    # NO assertion. This "covers" every line of apply_discount()
    # and would still pass if apply_discount returned garbage.

That test executes the function, so coverage counts every line as covered — yet it asserts nothing and would pass no matter how broken apply_discount is. A codebase can show 90% coverage and be a minefield; another at 70% with sharp, meaningful assertions can be far safer. Chasing a coverage number also pushes teams to write easy tests for trivial code (getters, plumbing) to bump the percentage, while the gnarly edge cases — the ones that actually break — stay untested because they’re hard.

Use coverage as a floor and a spotlight (this file has zero tests — is that on purpose?), never as the definition of “well-tested.” The real question is the one this whole book keeps asking: if this behaviour broke, would a test catch it? No coverage percentage can answer that for you.

The architect’s lens

• Why does it exist? Because automation that depends on a human remembering to run it isn’t automation — it’s a manual step in disguise. CI removes the human from the loop so the suite runs on every change, making the confidence automatic rather than optional.
• What problem does it solve? It closes the gap between “a test exists” and “the test protects us.” It turns a checkout-build-test cycle into an enforced gate, so breakage is caught minutes after it’s written, by a neutral machine, for everyone equally.
• What are the trade-offs? You pay in speed discipline and maintenance: the suite must stay fast (or people route around it), deterministic (or the signal rots), and pruned (or it decays). CI infrastructure and flake-hunting are a real, recurring cost.
• When should I avoid it? Never avoid CI itself — but don’t put a slow, flaky, or approximate test into a blocking gate. Quarantine it or fix it first. A gate is only worth having if it’s trustworthy; a flaky gate is worse than no gate.
• What breaks if I remove it? The link between changing code and knowing it still works. Without CI, “it works on my machine” returns, merge hell returns, and the suite silently stops protecting anyone the first time someone’s in a hurry.

Check your understanding

1. A team has a thorough automated suite but only runs it locally before big releases. Using the book’s throughline, explain why this suite provides far less justified confidence than the same suite wired into CI.
2. Name the four common root causes of flaky tests, and give a concrete fix for each.
3. Why is configuring CI to automatically retry failed tests a worse response to flakiness than quarantining the offending tests? What does each do to the green-build contract?
4. A codebase reports 92% line coverage. Give two distinct reasons this number could still hide serious, untested risk.
5. Explain why “keep the suite fast” and “keep the suite deterministic” turn out to rely on the same underlying property of tests.

Show answers

1. Confidence that depends on a human remembering to run the suite isn’t justified — it’s conditional on attention that will eventually lapse (holidays, hurry, a forgetful teammate). CI ties the check to the act of changing the code: it runs on every push automatically, on a neutral machine, so the signal exists whether or not anyone remembered. The local-only suite protects nobody the one time it matters.
2. Shared state (two tests touch the same file/row/global) — fix with per-test isolation and fresh setup/teardown. Timing/races (sleep-and-hope) — wait for a condition, not a clock. Order dependence (a test relies on an earlier one) — make tests independent and randomise order to surface it. External dependencies (real APIs, network, live clock) — use test doubles and an injected/frozen clock.
3. Automatic retry hides flakiness: it turns intermittent failures into passes, so genuinely broken code can slip through, and it removes any pressure to fix the flake. Quarantine keeps the test visible and owned while pulling it out of the blocking gate. Retry corrupts the contract (red no longer means stop, because reds get silently retried away); quarantine preserves the contract by keeping the remaining gate honest and putting the flake on a fix deadline.
4. (a) Coverage counts whether a line ran, not whether its behaviour was asserted — a test with no assertions can cover code that’s completely broken. (b) The 8% uncovered (or the assertions that are weak/approximate) may be exactly the gnarly edge cases that actually break, while the 92% is easy plumbing padded to lift the number. Coverage is a spotlight for untested code, not a measure of confidence.
5. Both depend on test independence / isolation. Determinism needs each test to own its state so no test’s outcome depends on another’s leftovers or on order. Speed via parallelism and sharding needs the same thing: only independent tests can safely run at the same time or be split across machines. Fix isolation and you get both.