Integration Strategies: Big-Bang vs Incremental

The overview drew the boundary: unit tests prove one module works in isolation, integration tests prove two or more work together — that the call actually connects, the data actually crosses, the assumptions actually match. This page answers the next, very practical question: in what order do you wire the pieces up, and what do you test against the pieces that aren’t wired up yet?

That second half is the whole difficulty. In any real system, module A calls module B calls module C. If you want to test A today but B and C don’t exist yet — or exist but are slow, flaky, or destructive — you cannot just “run A.” You need a stand-in for the parts that are missing. The strategy you pick decides which stand-ins you must build, and — more importantly — how hard your bugs will be to find.

The core problem: fault localization

Here is the first principle that governs everything on this page. When an integrated test fails, you learn that something is wrong. What you actually want to know is where. The distance between those two — “a test failed” and “here is the broken line” — is called fault localization, and it is the single most expensive part of debugging integration failures.

Fault localization gets exponentially harder as the number of newly connected, untested modules in one test grows. Wire two fresh modules together and a failure has essentially one suspect boundary. Wire fifty together and a failure has hundreds of candidate boundaries, any combination of which could be the cause. This is the lens through which to judge every strategy below: does it keep the number of new suspects per failing test small?

Big-bang integration

The tempting approach: build every module, wire them all together at once, and run the whole assembled system. This is big-bang integration.

   ┌───┐  ┌───┐  ┌───┐  ┌───┐  ┌───┐
   │ A │  │ B │  │ C │  │ D │  │ E │      each built in isolation…
   └───┘  └───┘  └───┘  └───┘  └───┘
      └──────┴──────┴──────┴──────┘
                   ▼
        ┌─────────────────────┐
        │  wire ALL at once    │  …then connected in one step
        │  and run the system  │
        └─────────────────────┘
                   ▼
              ❌ a test fails
        Which of the ~10 new boundaries broke it?

It looks efficient — no scaffolding, no stubs, no phased plan, just assemble and go. It is a trap, and the reason is fault localization. When the assembled system fails (it will), every newly-created interface is a suspect at once. You have no prior run where A–B worked but A–B–C didn’t, so you cannot bisect the failure to a single boundary. You are debugging N modules and N-squared potential interactions simultaneously, on the first run, with no green baseline to compare against.

Big-bang also fails late. You get zero integration feedback until the last module lands, so a mismatch designed in on day one isn’t discovered until week ten — after every downstream decision has been built on top of it. The lesson is blunt: big-bang minimizes setup effort and maximizes debugging effort, and debugging is the expensive one.

It actually happened — Ariane 5 Flight 501, June 1996

The maiden flight of the European Ariane 5 rocket was lost about 37 seconds after launch. The proximate cause was a piece of inertial-reference software reused from Ariane 4: a 64-bit floating-point value was converted to a 16-bit integer, and the larger horizontal velocity of the new, more powerful rocket overflowed that integer. The deeper lesson is an integration one. The reused module had been verified thoroughly on its own; what was never tested was the module wired into the new vehicle’s flight envelope — the specific combination of old code and new operating conditions. A component that is correct in isolation is not correct once its callers change. That gap between “the unit passed” and “the assembly passed” is precisely what integration testing exists to close.

Incremental integration

The alternative is to add modules one at a time, testing after each addition. Every step introduces exactly one new module and (usually) one new boundary, so when a test goes red the suspect is obvious: the thing you just added.

   step 1:  [A]────test✓
   step 2:  [A]──[B]────test✓        one new boundary → one suspect
   step 3:  [A]──[B]──[C]────test✗   C (or the B–C wire) broke it

You always have a green baseline — the last passing configuration — so every failure bisects itself. The cost is scaffolding: because you are running a partial system, the modules that aren’t there yet must be faked. That fake is either a stub or a driver, and which one you need depends on the direction you grow the system. Which brings us to the two classic orders.

Stubs and drivers: the two kinds of stand-in

Two words, precisely defined, unlock the rest of the page. Think about a single module under test. It has things it calls (its callees, below it) and a thing that calls it (its caller, above it). To run it in a partial system you may have to fake either side.

• A stub is a fake callee. It stands in for a lower module the code-under-test calls, and returns canned answers. Stubs let you test a module before its dependencies exist.
• A driver is a fake caller. It stands in for a higher module that would invoke the code-under-test, feeding it inputs and checking outputs. Drivers let you exercise a module before the thing that would use it exists.

              ┌──────────┐
   driver ───►│  MODULE   │  a driver calls INTO the module (fake caller,  above)
   (fake      │  UNDER    │
    caller)   │  TEST     │───► stub  a stub is called BY the module (fake callee, below)
              └──────────┘

Mnemonic: a driver drives the module from above; a stub is a stubby placeholder below. The strategy you choose is, in large part, just a decision about which of these two you will spend your time writing.

Top-down integration

Top-down starts at the top of the call tree — the high-level modules that orchestrate everything — and works downward. Because the lower modules a top-level module calls don’t exist yet, you replace each one with a stub.

        [ UI / controller ]         ← test this first
              │  calls
        ┌─────┴─────┐
     (stub)      (stub)             ← lower modules faked until built

Then you replace stubs with the real modules one layer at a time, retesting as you descend. The payoff: the high-level control flow and user-facing behavior are exercised early, so architectural and interface mistakes near the top surface on day one instead of at the end. The cost: you write and maintain a lot of stubs, and — the classic weakness — the low-level modules that do the real work get tested last, so a deep bug in a leaf module hides until the final layers land.

Bottom-up integration

Bottom-up is the mirror image. Start at the leaves — the low-level utility modules with no dependencies of their own — and build upward. Because nothing calls them yet, you write a driver to invoke each one.

     (driver)   (driver)           ← fake callers exercise the leaves first
         │           │
     [ leaf ]    [ leaf ]          ← test these first
         └─────┬─────┘
        [ higher module ]          ← added once leaves are trusted

The payoff is the inverse of top-down: the foundational, hardest-to-fake logic is proven first, on solid ground, before anything is built on top of it. The cost is also the inverse: the top-level control flow and overall user experience aren’t exercised until the very end, so a flaw in how the whole thing hangs together shows up late. And you trade stubs for drivers — the scaffolding just moves to the other side of each module.

The repo’s kvlite store is a clean bottom-up example. Its server tests treat the on-disk key-value store as a trusted leaf and drive it through a socket, one command at a time:

// rust/kvlite/tests/server.rs — a test driver exercises the leaf store over a connection.
let (db, _) = Db::open(&path).unwrap();
// the test IS the driver: it feeds commands in and asserts on the replies
assert_eq!(round_trip(&mut reader, &mut writer, "SET name ada"), "OK");
assert_eq!(round_trip(&mut reader, &mut writer, "GET name"), "ada");

That round_trip helper is a hand-written driver: no higher module calls the store yet, so the test itself plays the caller. Prove the leaf, then build the request-router on top of it.

Sandwich / hybrid integration

Pure top-down and pure bottom-up each leave one end of the system untested until last. The obvious fix is to do both at once. Sandwich (or hybrid) integration splits the system into three layers: a top layer integrated top-down, a bottom layer integrated bottom-up, and a middle “target” layer where the two efforts meet.

   top layer      ──► integrated top-down  (stubs downward)
   ───────────────────────────────────────
   target layer   ──► where the two meet
   ───────────────────────────────────────
   bottom layer   ──► integrated bottom-up (drivers upward)

The benefit is parallelism and balance: the UI team drives downward with stubs while the platform team builds upward with drivers, and both critical ends get early coverage. The cost is that you now write both stubs and drivers, and the middle layer, where the sandwich closes, gets the least isolated testing — it is exercised only once both halves arrive, which can hide its bugs the way big-bang hides everything.

Under the hood — how continuous integration reframes the question

The four strategies above are artifacts of a world where modules were built over months and you had to decide an order because the whole system was never simultaneously buildable. Modern continuous integration quietly dissolves much of that framing. When every developer merges to a shared main branch many times a day and an automated suite runs on every merge, the system is integrated continuously and in small increments by construction — that is literally what “continuous integration” means.

CI doesn’t repeal the physics of fault localization; it industrializes the incremental principle. Because each merge is tiny and the whole suite runs against it, a red build points at the handful of lines that just changed — the ultimate small-suspect-set. The strategic question shifts from “top-down or bottom-up over the next quarter?” to “what do we stub or run for real on every commit?”, which the next pages take up: contract testing for cheap, fast agreement between services, and real vs fake dependencies for when a stub is honest enough and when only the real thing will do.

By the numbers — suspects per failing test

The whole argument reduces to counting suspects. With n mutually-calling modules, the number of pairwise interfaces that could be at fault is roughly n(n-1)/2.

  big-bang, n = 10 modules wired at once:
      suspect interfaces ≈ 10 × 9 / 2 = 45   ← all live on the first failing run

  incremental, add the 10th module:
      new suspect interfaces = 1             ← the one boundary you just added
      (the other 44 already have green baselines)

  ratio of first-failure suspects: 45 : 1

The figures are illustrative, not measured — but the shape is real and it is the entire case for incremental integration: you are choosing between debugging one boundary and debugging all of them at once.

The architect’s lens

• Why does it exist? Because a system is more than its modules — the connections between them carry their own bugs (mismatched types, wrong units, broken contracts) that no isolated unit test can see. An integration strategy is a plan for surfacing those connection bugs in an order you can actually debug.
• What problem does it solve? Fault localization. A deliberate order (and the stubs/drivers it implies) keeps the number of new suspects per failing test small, turning “something in this 50-module system is broken” into “the boundary I just wired is broken.”
• What are the trade-offs? Incremental strategies trade scaffolding effort (stubs for top-down, drivers for bottom-up, both for sandwich) for debugging effort. Top-down proves user-facing flow early but tests the real work last; bottom-up proves the foundations first but tests the whole-system flow last. Big-bang writes no scaffolding and pays for it in undebuggable failures.
• When should I avoid it? Elaborate top-down/bottom-up sequencing is largely obsolete for teams practicing genuine continuous integration, where small, frequently-merged increments give you incremental integration for free. Don’t impose a heavyweight phased plan on a system that is already integrated on every commit.
• What breaks if I remove it? Remove strategy entirely and you get big-bang by default: late feedback, a green baseline that never existed, and every interface a suspect on the first failing run. The bugs don’t go away — they just all arrive at once, at the worst possible time, with the least information to fix them.

Check your understanding

1. In one sentence, why does big-bang integration make fault localization so hard, even when every individual module has passing unit tests?
2. Define a stub and a driver in terms of caller and callee. Which one does top-down integration require, and which does bottom-up require?
3. A team integrates top-down. Which parts of the system get exercised earliest, and which real bug is most likely to hide until the very end?
4. Sandwich integration is described as balancing top-down and bottom-up. What is the price of that balance — what do you have to write, and which layer gets the weakest isolated testing?
5. How does continuous integration change the practical relevance of choosing between top-down and bottom-up ordering?

Show answers

1. Because it wires many untested boundaries together at once with no prior passing configuration to compare against — so a failure has every new interface as a simultaneous suspect, and passing unit tests say nothing about whether the connections between modules are correct.
2. A stub is a fake callee (a stand-in for a lower module the code-under-test calls, returning canned answers); a driver is a fake caller (a stand-in for a higher module that invokes the code-under-test). Top-down needs stubs (the lower modules don’t exist yet); bottom-up needs drivers (nothing calls the leaves yet).
3. Top-down exercises the high-level control flow and user-facing behavior earliest. The likely late-hiding bug is a defect in a low-level leaf module, because the real low-level work is integrated and tested last.
4. You must write both stubs (for the top-down half) and drivers (for the bottom-up half). The middle/target layer, where the two efforts meet, gets the least isolated testing — it is only exercised once both halves arrive.
5. CI integrates the system continuously in small increments by construction, so you get incremental integration (and tiny suspect sets) for free on every merge. The heavyweight question of top-down vs bottom-up sequencing mostly dissolves; the live question becomes what to stub versus run for real on each commit.