The ROI Test — What to Automate (and What Not To)

The previous page made the case for automation as confidence at scale — a suite you can re-run for free on every commit. But “you can automate it” is not the same as “you should.” An automated test is not a one-time purchase. It is a small program you write once and then own forever, and forever is expensive.

So before we build any suite, we have to answer the most important — and most skipped — question in test automation: which tests actually earn their keep? Automate the wrong things and you get a slow, flaky, high-maintenance codebase-of-tests that erodes trust faster than it earns it. This page gives you a model for deciding, deliberately, one test at a time.

Automation is an investment, not a checkbox

Every automated test has a cost curve and a value curve, and they meet the world at different times.

   AUTHORING          MAINTENANCE                 VALUE
   (one-time)         (every change, forever)     (every run, forever)
   ┌──────────┐       ┌──────────────────────┐    ┌──────────────────┐
   │ design   │       │ update when code      │   │ catches a        │
   │ write    │  ──►  │   or UI changes        │──►│  regression       │
   │ debug    │       │ chase flakiness        │   │ before a human    │
   │ make it  │       │ refactor with the app  │   │  would have to    │
   │  green   │       │ re-triage failures     │   │  run it by hand   │
   └──────────┘       └──────────────────────┘    └──────────────────┘

The mistake is to see only the left box. Teams estimate the authoring cost, decide it is affordable, and never price the middle box — the one that never stops. A test suite is not something you have; it is something you keep. And you keep paying for it in the same currency you pay for any other code: engineer time.

The ROI formula, in plain terms

Here is a mental model — not an exact accounting, but a way to make the trade-off visible. For any candidate test:

   value  =  (manual_cost × run_frequency × test_lifespan)
   cost   =  authoring_cost  +  maintenance_cost

   worth automating  when  value > cost
   i.e.  (manual_cost × run_frequency × test_lifespan)
            − (authoring_cost + maintenance_cost)  >  0

Read each term as a question:

manual_cost — how long does it take a human to run this check once, by hand? A one-line assertion is cheap to eyeball; a twelve-step checkout flow is not.
run_frequency — how often will this check need to run? On every commit? Nightly? Once, ever?
test_lifespan — how long will this check stay relevant before the feature is deleted or rewritten? Stable core logic lives for years; a promo banner lives for a fortnight.
authoring_cost — how hard is it to write the automated version and get it reliably green? (The “reliably” is where the hours hide.)
maintenance_cost — how often will this test break because the code changed, not because a bug appeared? A test wired to volatile internals or shifting UI has a high, recurring maintenance cost.

The insight is that value is a product — manual_cost × run_frequency × lifespan. If any one factor is near zero, the whole value term collapses, no matter how large the others are. A check that runs only once (run_frequency ≈ 1) or covers a feature that dies next sprint (lifespan ≈ 0) has almost no automation value even if it is expensive to run by hand.

Rough, illustrative figures — plug in your own, the shape is what matters:

   Candidate A: login on the critical path
     manual_cost    = 2 min/run  (0.033 h)
     run_frequency  = 40 runs/day × 250 days = 10,000 runs/yr
     lifespan       = 3 yr
     → manual value ≈ 0.033 × 10,000 × 3 ≈ 1,000 hours saved
     authoring      = 4 h
     maintenance    = 1 h/yr × 3 = 3 h
     ROI = 1,000 − 7 ≈ +993 h        ► automate, obviously

   Candidate B: one-off data-migration sanity check
     manual_cost    = 15 min (0.25 h)
     run_frequency  = 1 (you run the migration once)
     lifespan       = irrelevant — feature is gone after
     → manual value ≈ 0.25 × 1 ≈ 0.25 hours
     authoring      = 3 h    maintenance = 0
     ROI = 0.25 − 3 ≈ −2.75 h         ► do it by hand, once

Same engineers, same tools — opposite decisions. The formula did not tell you login is important; it told you login is run ten thousand times a year for three years, which is why automating it is not close.

Strong candidates — automate these

The formula favours tests where value compounds and cost stays low. In practice, that is:

Stable, high-value paths. The flows that, if broken, mean lost revenue or a broken product: login, checkout, payment, signup, the core create/read/update/delete of your main entity. High manual_cost to verify carefully, high run_frequency because you never want them broken, long lifespan because they rarely go away.
Deterministic logic. Pure functions and business rules — tax calculation, discount tiers, date math, parsing, validation. Same input, same output, every time. Cheap to author, near-zero maintenance, and exactly the kind of thing humans get subtly wrong. This is the sweet spot of the automation pyramid’s base (which the next page revisits).
Regression-prone areas. Anywhere a bug has already bitten you once. A test written the moment a bug is fixed is worth its weight — it has proven demand. It pins the behaviour so the same defect cannot silently return.
Anything run on every commit. If a check belongs in the CI gate, its run_frequency is enormous by definition. That single factor usually clears the ROI bar on its own.

A useful test of a good candidate: would you be annoyed to check this by hand again? And will you have to? If both are yes, automate it.

Poor candidates — leave these to a human

Just as important is knowing what to not automate. Forcing these into a suite is how you end up maintaining a liability.

One-off checks. A single data migration, a quick “did this deploy land?” sanity look. run_frequency ≈ 1 kills the value term. Do it by hand and move on.
Rapidly-changing UI. Layout, wording, and DOM structure that shift every sprint have a punishing maintenance_cost: the test breaks constantly, and almost always because the code changed as intended, not because a bug appeared. A test that cries wolf on every legitimate change is worse than no test.
Things that are flaky by nature. Checks that depend on real network timing, third-party services, animation, or wall-clock races. If a test cannot be made reliably green, its authoring_cost is effectively infinite, and each red run costs triage time whether or not there is a real bug. (We will fight this head-on in Running in CI.)
What a human eye validates far cheaper. Visual polish, layout aesthetics, “does this feel right?” UX judgement, copy tone. A person confirms these in seconds; an automated equivalent (pixel diffing every screen) is expensive, brittle, and still misses the actual question — is it good? Machines check presence; humans judge quality.

   AUTOMATE                          KEEP MANUAL / EXPLORATORY
   ─────────                         ─────────────────────────
   deterministic logic               "does this feel right?"
   critical path, run always         one-off / run-once checks
   known past regressions            rapidly-shifting UI
   fast, stable, repeatable          inherently flaky / timing
   → the KNOWN                       → the UNKNOWN & the SUBJECTIVE

Automate the known; explore the unknown

There is a category error lurking here. Automation is very good at one thing: re-checking, tirelessly and identically, something you already knew to look for. Every automated test encodes a question you thought to ask in the past. It can never ask a question nobody wrote down.

That is precisely what exploratory testing is for — a human wandering the product with intent, forming hypotheses, following hunches, noticing the thing that looks off in a way no assertion anticipated. Automation cannot replace it, because you cannot automate a question you have not thought of yet.

So the division of labour is not “automate everything eventually.” It is:

   the KNOWN     ──►  automate it   (regression, repeatability, scale)
   the UNKNOWN   ──►  explore it    (a curious human, unscripted)

A healthy strategy does both. Automation frees your humans from re-running the known so they can spend their scarce attention exploring the unknown — which is the only place new bugs are ever found.

In August 2012, the trading firm Knight Capital deployed new code to its servers. By most accounts a piece of old, dormant logic was reactivated on a subset of machines, and within roughly 45 minutes the system sent a flood of unintended orders — a loss commonly cited as around $440 million, enough to nearly destroy the firm.

The testing lesson is not “they lacked tests.” It is that the deployment path itself — how code reaches which servers, and what happens when a flag flips on a partial rollout — was a high-value, regression-prone, run-on-every-deploy scenario: a textbook strong candidate for automated verification. The known failure modes of deploying were exactly the kind of thing a suite should have guarded, and a human eyeballing a dashboard could never have caught in 45 minutes. (Specifics are reconstructed from post-mortems; treat the numbers as approximate — the lesson stands.)

Under the hood — maintenance is a codebase you own

The term that ruins more suites than any other is maintenance_cost, because it is invisible at authoring time. A large automated suite is, quite literally, a second codebase — often as large as the application it tests. It has functions, abstractions, dependencies, and bugs. It must be:

refactored when the app is refactored, or it rots into irrelevance;
debugged when it goes red, and you must decide every time whether the red is a real bug or a broken test (a distinction that itself costs time);
kept fast, or engineers start skipping it and its run_frequency — its whole value — silently drops to zero.

This is why “automate everything” is not a virtue. Every test you add is a permanent liability on the maintenance ledger as well as an asset on the value ledger. A smaller suite of high-ROI tests that everyone trusts beats a giant suite half of which is muted, flaky, or ignored. The goal is not maximum tests; it is maximum justified confidence per hour of upkeep.

The architect’s lens

Why does it exist? Because automation capacity is finite and every automated test is owned forever. The ROI test exists to spend that finite budget on the tests that repay it, instead of automating reflexively until the suite collapses under its own maintenance weight.
What problem does it solve? It stops two failure modes at once: under-automating the stable, high-frequency paths that a human should never re-check by hand, and over-automating the one-off, subjective, or fast-changing checks that a human validates far more cheaply.
What are the trade-offs? The model is a heuristic, not accounting — the inputs (manual_cost, lifespan, maintenance_cost) are estimates, and estimating them well is a skill. Applied too literally it can under-value hard-to-price things like the confidence a green suite gives a nervous release.
When should I avoid it? Never skip the thinking, but don’t ceremonialise it — you don’t need a spreadsheet per test. And don’t let a marginal-ROI verdict veto a test the law requires (a compliance or safety check gets automated regardless of the sums).
What breaks if I remove it? You automate by vibe. The suite fills with brittle UI tests and one-off checks, flakiness rises, trust falls, engineers start ignoring red — and you inherit all the cost of automation with a shrinking share of its value.

Check your understanding

State the ROI model in your own words. Why is value written as a product of three factors rather than a sum, and what does that imply if any one factor is near zero?
A teammate wants to automate a pixel-perfect check of the marketing homepage’s hero image, which the design team restyles roughly every two weeks. Walk through the formula and give your recommendation.
Why is a test written the moment a bug is fixed an unusually strong automation candidate? Name the term in the formula it scores highest on.
“We should automate everything eventually” — using the maintenance argument, explain why this is a costly goal rather than an aspirational one.
Automation and exploratory testing are not substitutes. Which one covers the known and which covers the unknown, and why can automation never replace the other?

Show answers

Roughly: value = manual_cost × run_frequency × test_lifespan, and you automate when value exceeds authoring_cost + maintenance_cost. It is a product because the factors compound — a check that is expensive to run by hand is still worthless to automate if it runs once or covers a feature that dies next week. If any single factor is near zero, the whole value term collapses, regardless of how large the others are.
manual_cost is low (a designer eyeballs it in seconds) and, worse, it validates subjective visual quality a human judges far more cheaply. Meanwhile the fortnightly restyle gives it a high maintenance_cost and short effective lifespan, so the test would break constantly on intended changes. Verdict: do not automate — leave it to a human eye; a rare structural smoke check (“image element loads at all”) is the most one might automate.
Because it has proven demand: the bug already happened, so this is a regression-prone area — it scores highest on the fact that it guards a known past failure. Automating it pins the behaviour so the exact same defect cannot silently return, and it typically also runs on every commit (high run_frequency).
Every automated test is a permanent entry on the maintenance ledger, not just the authoring ledger — it must be refactored, debugged, re-triaged, and kept fast forever. “Automate everything” maximises that recurring cost and produces a giant, partly-flaky suite people stop trusting. The real goal is maximum justified confidence per hour of upkeep, which favours a smaller, high-ROI, trusted suite.
Automation covers the known — it tirelessly re-checks questions you already thought to ask. Exploratory testing covers the unknown — a curious human forming new hypotheses unscripted. Automation can never replace exploration because it can only assert questions someone already wrote down; it cannot ask a question nobody has thought of yet, which is where new bugs are found.