Error Guessing and Experience-Based Testing

The five techniques before this one — equivalence partitioning, boundary value analysis, decision tables, state-transition testing, and pairwise — all share one shape. You read the specification, model it as classes or edges or transitions or factors, and mechanically derive cases. Anyone following the same recipe on the same spec gets roughly the same suite. That is their strength: the coverage is countable, and it does not depend on who holds the pen.

This page is about the technique that breaks that pattern on purpose. Error guessing — the leading edge of experience-based testing — does not start from the spec. It starts from the tester’s memory of how software actually fails: the empty string that was never handled, the emoji that broke the database, the second click that fired the payment twice. Where the formal techniques cover the ground the spec describes, error guessing goes hunting for the ground the spec forgot to describe. It is the deliberate, disciplined use of intuition — and the whole point of this page is to make it disciplined rather than random.

Why a spec-blind technique earns its place

Every specification-based technique inherits the spec’s blind spots. If the requirement never mentions what happens when the input is 10 MB of whitespace, no partition, boundary, or table will ever produce that case — the model was built from a document that was silent on it. That silence is not rare; it is the normal condition of every spec ever written. Requirements describe the intended behaviour, and bugs live overwhelmingly in the unintended behaviour nobody wrote down.

Error guessing attacks exactly that gap. It asks a different question from the formal techniques:

Specification-based:  "What does the spec say should happen? Test each case."
Experience-based:     "Where do systems like this usually break? Go there."

The knowledge it draws on is real and transferable. A tester who has shipped a dozen web forms knows that the “name” field will one day receive O'Brien and that the apostrophe will find an unescaped SQL string somewhere. No requirement predicts that bug. Experience does. Error guessing is the mechanism for turning that hard-won pattern memory into concrete test cases before the customer does it for you.

There is a common misreading worth killing early: that “experience-based” means “unstructured” or “just poke at it.” It does not. A random poke has no memory and no direction. Error guessing is directed — every guess traces back to a specific past failure or a specific class of failure the tester has seen before. The intuition is the index into a library of real defects, not a substitute for thinking. That is precisely why the rest of this page is checklists, attack tables, and a feedback loop rather than a pep talk about “thinking creatively.” The creativity is in choosing which known failure mode applies here; the discipline is in having the known failure modes written down at all.

The usual suspects: a standing checklist

Intuition is unreliable when it is vague and reliable when it is a checklist. Experienced testers do not “feel” their way to good inputs — they carry a mental list of the inputs that have historically broken things, and they run down it for every field, endpoint, and parameter. Writing the list down does two things: it makes a junior tester’s coverage approach a veteran’s, and it stops even the veteran from forgetting a category under deadline pressure. Keep this list where you can see it:

• Empty input — the "", the empty list, the zero-length upload, the request with no body. Code that assumes “there is always at least one” breaks on the first element that isn’t there.
• Null / absent — a missing optional, a None, a JSON key that simply isn’t present versus one present but set to null. The most infamous crash in software has this shape.
• Huge input — a 5,000-character name, a 2 GB file, a list with a million entries. Buffers, timeouts, and memory limits all have a ceiling nobody documented.
• Special characters — quotes ' ", angle brackets < >, backslashes, semicolons, %, null bytes. These are the raw material of SQL injection, XSS, and path traversal, and they hit parsers that assumed friendly input.
• Duplicates — the same item added twice, the same key inserted twice, the same request submitted twice (the double-click that charges the card twice). Uniqueness assumptions are rarely tested against their own violation.
• Whitespace — a leading space, a trailing tab, a value that is only spaces, \r\n line endings from a Windows paste. " admin" and "admin" are different strings but often the same user.
• Encoding — UTF-8 multi-byte characters, emoji, right-to-left text, the four-byte characters that break utf8 (three-byte) MySQL columns. “It worked with ASCII” is where a whole class of production incidents begins.

Notice these are domain-independent. You apply the same seven to a login form, a payment API, and a config file parser, because they describe how the machine breaks, not what the feature does.

The checklist is a starting frame, not a ceiling. Each category fans out into a family of concrete values, and a good tester runs the whole family, not one token per row:

Category          Concrete values to actually try
────────────────  ──────────────────────────────────────────────────
Empty             ""  ·  []  ·  {}  ·  a request with no body at all
Null / absent     null  ·  key missing entirely  ·  None vs "None"
Huge              length = max+1  ·  10 MB string  ·  10^6-element list
Special chars     '  "  <  >  \  ;  %  --  \0  ·  ../  ·  ${...}
Duplicates        same key twice  ·  same request twice  ·  same file
Whitespace        " x"  ·  "x "  ·  "   "  ·  "\t"  ·  "\r\n"  ·  NBSP
Encoding          "café"  ·  "😀"  ·  RTL "‏..."  ·  4-byte vs 3-byte UTF-8

Two of these rows deserve a second look because they masquerade as ordinary text. A non-breaking space (NBSP, U+00A0) looks identical to a normal space in a UI but is a different byte, so "admin" with a trailing NBSP passes a naive “no trailing space” trim and slips through. And a four-byte emoji is valid UTF-8 that a three-byte MySQL utf8 column silently truncates or rejects — the input is legal, the storage layer is not, and the failure surfaces far from the field that accepted it.

Fault attacks: think like an adversary

Error guessing has a sharper, more deliberate cousin sometimes called a fault attack. Instead of asking “what input might be unusual?”, you ask “what has the developer assumed, and how do I violate that assumption on purpose?” You stop being a user and start being an adversary against the code’s own beliefs.

Every piece of code carries implicit assumptions. A fault attack enumerates them and attacks each one:

Developer's assumption           Fault attack that violates it
──────────────────────────────   ─────────────────────────────────────
"The file exists"                delete it between the check and the read
"The number fits in the field"   send 2^63, or -1, or NaN
"They fill fields in order"      submit step 3 before step 1
"The list is non-empty"          send []
"The disk has space"             fill it, then write
"The response comes back"        cut the network mid-request
"Two users won't collide"        fire both writes at the same key at once

That last row is a real, testable attack. The kvlite server in this repo answers SET/GET/DEL over a socket; a fault attacker does not test those one at a time — they fire many at once and check the invariant still holds:

// Attack the assumption "only one client touches a key at a time".
// Spawn N threads that all SET/GET the same key; the count must stay exact.
let handles: Vec<_> = (0..8).map(|i| {
    let db = db.clone();
    thread::spawn(move || {
        for _ in 0..1000 {
            db.set(format!("k{i}"), "v".into());
            assert_eq!(db.get(&format!("k{i}")), Some("v".to_string()));
        }
    })
}).collect();
for h in handles { h.join().unwrap(); }
// No lost writes, no torn reads — the concurrency assumption held.

The formal techniques would never generate this test, because no partition or boundary table has a dimension for “at the same time.” A fault attack does, because it starts from the assumption, not the spec.

A practical way to run a fault attack is to read the code (or the design) and write down every sentence that begins with “we can assume” or “this will always.” Each such sentence is a target. The developer who wrote “we can assume the config file is present” has handed you a test: rename the file and run. The one who wrote “the queue will always drain faster than it fills” has handed you a load test. Assumptions are not bugs — but every one is an untested claim, and untested claims are where the nastiest, least-anticipated failures wait. The fault attacker’s job is to make each claim earn its keep by trying to break it on purpose.

It actually happened — the double-submit that fired live

On a well-known trading firm’s deployment in 2012, new order-routing code went live alongside an old, dormant code path that a reused feature flag accidentally re-activated on some servers. The system began sending a flood of unintended orders — buying and selling at a rate no human had authorised — and within roughly 45 minutes the firm had accumulated losses reported in the hundreds of millions of dollars, enough to nearly destroy it.

The details are specific to deployment, but the testing lesson is pure error guessing. “What happens when an old code path and a new one run at the same time?” and “what if this action repeats far more often than we assumed?” are exactly the adversarial, assumption-violating questions a fault attack asks — and exactly the questions no specification-based table, built from the intended behaviour, would ever produce. The nastiest failures live in the interactions nobody wrote a requirement for.

Experience-based versus specification-based: different gaps

It is tempting to rank these approaches. Don’t — they cover different gaps, and a suite needs both.

                Specification-based            Experience-based
                (partitions, boundaries,       (error guessing,
                 tables, state, pairwise)       exploratory)
────────────    ──────────────────────────     ─────────────────────────
Starts from     the written spec               the tester's defect memory
Coverage        measurable, repeatable         unmeasurable, tester-dependent
Finds           bugs in described behaviour    bugs in forgotten behaviour
Reproducible    same suite for anyone          varies by who runs it
Blind to        the unwritten cases            nothing systematic — by design

A specification-based suite gives you a floor you can count and hand to anyone. An experience-based session gives you a reach into the cases the floor was never built to hold. Run only the formal techniques and you ship the O'Brien bug; run only exploratory testing and you have no evidence you covered the documented requirements at all. The two are complements, and the combining-techniques page is about weaving them into one plan.

A good sequencing heuristic falls out of the table: use the specification-based techniques to build the baseline suite that must pass on every release, then spend a bounded, time-boxed exploratory session hunting for what the baseline missed. The formal work is the standing army that holds the documented perimeter; the exploratory session is the raid that finds new ground. Do the formal work first so the exploratory time is spent on genuinely new territory rather than re-discovering cases a partition table would have handed you for free.

The limits: unmeasurable and tester-dependent

Be honest about what error guessing cannot do, because its weaknesses are the mirror image of the formal techniques’ strengths.

• Coverage is unmeasurable. There is no “we tested 100% of the error guesses” — the space of things that might break is unbounded and undocumented. You cannot report a completion percentage, so you cannot use it as an exit criterion on its own.
• It depends entirely on the tester. Two testers produce two different suites; a junior tester’s guesses and a 15-year veteran’s guesses are not interchangeable. The technique’s yield rides on experience that does not distribute evenly across a team.
• It is not repeatable by construction. Run it twice and you get different cases — which is fine for finding bugs but useless as the stable regression suite you re-run every release.

The conclusion is not “avoid it.” It is: error guessing supplements systematic design; it never replaces it. Use it to find the bugs the spec forgot — then, crucially, do the next step.

A useful mental model: the specification-based techniques build a fence around the ground the spec describes, and you can measure exactly how much of that ground the fence encloses. Error guessing is a scout who walks outside the fence, into the terrain the map never showed, and reports back where the ground gives way. You cannot measure how much of the wilderness the scout covered — but everything the scout finds and brings back can be added to the map, and then a new, larger fence can enclose it. Scouting without fencing gives you no defensible perimeter; fencing without scouting means the map never grows. You want both, in that order, on a loop.

Feed guesses back into the systematic techniques

A guess that catches a bug once is a lucky shot. A guess folded back into a formal technique becomes a permanent, repeatable case that runs forever. This feedback loop is what turns experience-based testing from a one-off hunt into lasting coverage — and it is the single most valuable habit in this page.

The move is mechanical. When an error guess finds a bug, ask which formal technique should have caught it, and add the case there:

One-off guess that hit               Fold it back into…
──────────────────────────────       ─────────────────────────────────
"emoji broke the name field"    ──►  a new equivalence class: "multi-byte
                                     Unicode" alongside ASCII / empty / long
"$100.00 wasn't handled"        ──►  a boundary value on the price field
"double-click charged twice"    ──►  a decision-table rule / a state guard
                                     on the checkout state machine
"only-whitespace username"      ──►  an invalid equivalence class + a
                                     boundary at "1 non-space char"

After the fold-back, the case is no longer dependent on anyone remembering the trick. It sits in a partition or a table, gets a stable identifier, and re-runs on every build like any other systematic test. The exploratory session found the gap; the formal technique now guards it. Over time, a team that does this consistently is teaching its systematic suite everything its testers have learned the hard way.

Under the hood — turning a hunch into a repeatable assertion

The fold-back is not just a filing habit; it is a change in what kind of artifact the bug lives as. During exploration the catch exists only as a story in someone’s head — “I typed an emoji and the profile card blew up.” That story cannot be re-run. Encoding it as a real assertion is what makes it durable:

# The one-off exploratory catch, now a permanent regression case.
# It joins the "display name" equivalence classes as the "multi-byte" member.
def test_display_name_accepts_multibyte():
    resp = create_user(display_name="Ada 😀 Lovelace")
    assert resp.status == 201
    assert render_profile_card(resp.user_id).ok   # the card that crashed

Once written, that assertion has three properties the hunch never had: it is named (searchable, referenceable in a bug tracker), it is automatic (the CI job runs it without a human present), and it is falsifiable (it goes red the instant the bug regresses). A hunch has none of those. The whole value of the fold-back is the promotion from the first kind of artifact to the second — from a war story to a red-green test.

There is a subtler benefit too. When you place the new case inside an existing partition set, you are also forced to name the partition it belongs to (“multi-byte Unicode” alongside “ASCII”, “empty”, “over-length”). That act of naming often reveals neighbouring cases you had not tried — if multi-byte matters, what about right-to-left multi-byte, or a mix of scripts? The formal structure the guess lands in generates further guesses. This is why the loop compounds: each fold-back both guards one bug and widens the model that will catch the next.

By the numbers — where do error guesses come from?

Good error guessing is a checklist plus history, not magic. A concrete way to build the guess list for one field:

Field under test:  "display name" (a public username)

Standing checklist (domain-independent):   7 categories
  empty · null · huge · special-chars · duplicates · whitespace · encoding

Past-defect history for THIS product:      +N cases
  e.g. bug #4412: "emoji in name crashed the profile card renderer"
       bug #5109: "trailing space let two users register 'admin' twice"

Guesses generated:                         7 categories × relevant values
                                           + every historical defect re-run

Fold-back after the session:               each catch → 1 permanent
                                           partition / boundary / table rule

The list is finite and reusable. The history compounds: every incident this product survives should add a line, so the same class of bug can never ship twice.

The architect’s lens

• Why does it exist? Because every specification-based technique inherits the spec’s blind spots, and bugs live overwhelmingly in the behaviour the spec never described. Error guessing is the deliberate use of tester experience to reach that unwritten ground.
• What problem does it solve? It finds the nasty, high-yield inputs — empty, huge, special-character, duplicate, whitespace, encoding, assumption-violating — that no partition, boundary, or table would ever generate, because those inputs come from defect history, not from requirements.
• What are the trade-offs? Its reach is real but its coverage is unmeasurable and tester-dependent, and it is not repeatable by construction — so it cannot serve as an exit criterion or a stable regression suite on its own.
• When should I avoid it? Never as your only technique, and never where you need a countable, auditable coverage claim — reach for the specification-based techniques there. Use error guessing as the supplement that fills their gaps, not the foundation.
• What breaks if I remove it? Your suite tests only what the spec described and ships the classic production crashes — the apostrophe, the emoji, the double-click, the empty upload — because nothing in a spec-derived model was ever looking for them.

Check your understanding

1. Why can no specification-based technique (partitioning, boundaries, tables, state, pairwise) generate a test for “10 MB of whitespace in the name field,” and why can error guessing?
2. List the seven “usual suspects” on the standing checklist, and explain why the list is domain-independent.
3. What distinguishes a fault attack from ordinary error guessing? Give one developer assumption and the attack that violates it.
4. Name the two structural limits of error guessing that are the mirror image of the formal techniques’ strengths, and state the one-sentence conclusion they lead to.
5. Describe the fold-back loop: when an error guess catches a bug, what do you do next, and why does it matter?

Show answers

1. The formal techniques are all derived from the specification, so if the spec is silent about whitespace-only or oversized input, the model built from it has no dimension that produces such a case. Error guessing starts not from the spec but from the tester’s memory of how software actually fails, so it can target inputs the requirements never mentioned.
2. Empty, null, huge, special characters, duplicates, whitespace, encoding. The list is domain-independent because it describes how the machine breaks — buffers, parsers, uniqueness assumptions, string handling — not what any particular feature does, so the same seven apply to a login form, a payment API, and a config parser alike.
3. A fault attack is targeted and adversarial: instead of asking “what input is unusual?”, it enumerates the code’s implicit assumptions and deliberately violates each one. Example: the developer assumes “the file exists,” and the attack deletes the file between the existence check and the read (any assumption/violation pair from the table is acceptable).
4. Coverage is unmeasurable (no completion percentage, so it cannot be an exit criterion) and it is tester-dependent and not repeatable (two testers, or two runs, produce different suites, so it cannot be the stable regression suite). Conclusion: error guessing supplements systematic design — it never replaces it.
5. Ask which formal technique should have caught the bug, and add the case there — a new equivalence class, a boundary value, a decision-table rule, or a state guard. It matters because a folded-back case gets a stable identifier and re-runs on every build forever, turning a one-off lucky catch into permanent, repeatable coverage.