Functional Testing — Does It Do the Right Thing?

The overview laid out the whole landscape of testing. This page zooms in on the one kind that every other kind quietly assumes has already been done: functional testing — the check that the software actually does the right thing.

Everything else in this Part answers a different question. Performance testing asks is it fast enough? Security testing asks can an attacker break in? Accessibility asks can everyone use it? But before any of those matter, one question has to be answered first: given some input and some state, does the program produce the output the specification says it should? That is functional testing, and it is the backbone of earned confidence.

What functional testing actually is

Strip away the jargon and functional testing is a single loop:

   spec says:   given INPUT and STATE, the output should be EXPECTED
   ─────────────────────────────────────────────────────────────────
   1. set up STATE
   2. feed the software INPUT
   3. observe the ACTUAL output / behavior
   4. compare ACTUAL against EXPECTED
   5. pass if they match, fail if they don't

Notice what is and isn’t in that loop. We look only at observable behavior — the output, the return value, the response on the wire, the row that appears in the database, the pixel that changes on screen. We do not look at how fast it happened, how much memory it used, or how elegant the code was. Functional testing asks one thing: is the answer correct?

The word that carries all the weight is specification. A functional test is only as good as the “should” it encodes. If the spec says “analyze counts one line per non-blank log entry,” then a test that asserts total_lines == 6 on a seven-line-with-one-blank input is a functional test. If nobody ever decided what the right count is, you don’t have a test — you have a guess that happens to be green today.

A test is a claim about a specification

Look at a real assertion from logwise’s aggregation code:

#[test]
fn counts_levels_and_skips_blanks_and_malformed() {
    let report = analyze_sample(&Filter::default());
    let s = &report.summary;
    assert_eq!(s.total_lines, 6);   // 7 input lines minus 1 blank
    assert_eq!(s.parsed, 5);        // one line is malformed
    assert_eq!(s.malformed, 1);
    assert_eq!(s.counts[Level::Info.index()], 2);
    assert_eq!(s.counts[Level::Error.index()], 1);
}

Every one of those assert_eq! lines is a sentence from the spec written as code: blank lines don’t count; malformed lines are tallied separately; the histogram counts each level exactly. The test doesn’t care that analyze uses fold to build the histogram or filter to select matches. It cares that the numbers come out right. That is the essence of functional testing — the assertion is a promise about behavior, derived from a requirement, not from the implementation.

Functional testing is a concern, not a level

The levels of testing — unit, integration, end-to-end — describe how much of the system you exercise at once. Functional testing describes what you’re checking: correctness of behavior. These are two different axes. Functional testing cuts across every level.

                     WHAT you check  →  correctness of behavior (functional)
                                        speed (performance)
                                        safety (security) ...
   HOW MUCH you run
   ┌──────────────┬──────────────────────────────────────────────┐
   │ unit         │  analyze() returns matched == 2 for min WARN  │ ← functional
   │ integration  │  Db::open + SET + GET returns the value back  │ ← functional
   │ end-to-end   │  running `logwise app.log` prints "ERROR : 3" │ ← functional
   └──────────────┴──────────────────────────────────────────────┘

The same concern — did it produce the right answer? — shows up at all three levels, just at different scales:

• Unit level. analyze(SAMPLE.lines(), &filter, false) is called directly and its Report is inspected. One function, in-process, no I/O.
• Integration level. kvlite’s server test opens a real WAL-backed Db, binds a real TCP socket, and checks that a SET name ada lovelace followed by GET name returns VALUE ada lovelace — two components (storage and the wire protocol) proven correct together.
• End-to-end level. The logwise CLI test runs the built binary against a sample file and asserts the printed report contains ERROR : 3 — the whole program, from argument parsing to formatted output, as a user would run it.

All three are functional tests. They differ only in how much they wrap up in one check. So when someone says “we need more functional tests,” they haven’t told you where — that is a separate decision, covered in Choosing What a Change Needs.

Black-box vs white-box: two framings of the same check

There are two ways to derive a functional test, and they differ in what knowledge you’re allowed to use.

Black-box framing: you test against the specification and deliberately ignore the internals. You know only the contract — “GET on a missing key returns NIL” — and you write tests from that contract. You could swap the entire implementation and the tests should still pass, because they never looked inside. The logwise end-to-end tests are pure black-box: they run a compiled binary and read its stdout, with no visibility into fold or filter at all.

White-box framing: you use knowledge of the code to choose which cases to test. You read the implementation, notice there’s a branch for strict mode, an if raw.trim().is_empty() that skips blanks, and a < min comparison in the level filter — and you deliberately write a test that exercises each branch and boundary. The assertions still check observable behavior (that’s what makes them functional), but the selection of cases came from reading the code.

   BLACK-BOX                          WHITE-BOX
   ─────────                          ─────────
   "The spec says X.                  "The code has a branch here,
    Does the program do X?"            a boundary there — do they
                                       each behave correctly?"

   sees:  the contract                sees:  the source
   finds: missing features            finds: untested branches, edge cases
   risk:  misses hidden code paths    risk:  inherits the code's blind spots

The crucial insight: they check the same thing — correct observable behavior — they just disagree on how you pick the inputs. A mature test suite uses both. Black-box tests guarantee the contract holds regardless of implementation; white-box tests make sure no sneaky branch went unchecked. The level-filter boundary is a perfect white-box catch: the spec says “keep lines at this level or higher,” and the code compares line.level < min. A white-box tester sees the < and immediately asks — is a line exactly at min kept or dropped? — and writes:

let filter = Filter { min_level: Some(Level::Warn), ..Default::default() };
let report = analyze_sample(&filter);
assert_eq!(report.summary.matched, 2); // WARN itself is kept, plus ERROR above it
assert!(report.matches.iter().all(|l| l.level >= Level::Warn));

That boundary — is min inclusive? — is exactly the kind of off-by-one a black-box tester might skip and a white-box tester never does.

By the numbers — spec-derived assertions in logwise

The analyze function’s specification has a handful of independent rules. Each becomes one or more assertions:

   spec rule                              assertion
   ────────────────────────────────────   ──────────────────────────
   blank lines are ignored                total_lines == 6 (not 7)
   malformed lines counted separately     malformed == 1
   level filter is inclusive of `min`     matched == 2 for min=WARN
   pattern match is substring             matched == 1 for "request"
   --ignore-case folds case               "SERVER" matches "server"
   --strict aborts on first bad line      Err(Parse { line_no: 6 })

   6 spec rules  →  6+ targeted assertions, each independently falsifiable

The count that matters is not “how many tests” but “how many distinct rules of the spec have at least one test that would fail if the rule were violated.” Six rules, six checks: full functional coverage of this contract.

Under the hood — what makes an assertion trustworthy

A functional test earns trust only if it would actually fail when the behavior is wrong. Two failure modes quietly destroy that:

1. A test with no meaningful assertion. Running analyze(...) and only checking that it “didn’t panic” proves almost nothing — the numbers could all be wrong. The kvlite round-trip is trustworthy precisely because it asserts the exact reply string: assert_eq!(round_trip(..., "GET name"), "VALUE ada lovelace"). If storage silently dropped the value, that line goes red.

2. A test that asserts the implementation instead of the behavior. If you assert “the histogram was built with fold,” you’ll get a false failure the day someone rewrites it with a loop — even though the output is identical. Functional tests should assert what comes out, not how it was computed, so that a correct refactor stays green. That property — surviving refactors — is what makes functional tests the safety net for the regression work in the next page.

A good rule: for every assertion, ask “what real bug would this catch?” If the honest answer is “none,” delete it or sharpen it.

It actually happened — Mars Climate Orbiter, 1999

NASA’s Mars Climate Orbiter was lost on 23 September 1999 when it entered the Martian atmosphere at the wrong altitude and broke apart. The reported cause: one software component produced thrust values in pound-force seconds (imperial) while the component consuming them expected newton-seconds (metric). Each piece of software worked perfectly against its own idea of the spec — the numbers were internally consistent. What was missing was a functional test at the integration boundary asserting the end-to-end behavior: given a known trajectory input, does the combined system produce the correct course correction in agreed units? The lesson generalizes: unit-level functional tests can all pass while the contract between two units is silently wrong. Functional testing has to include the seams, not just the parts. (Details are summarized from the public mishap report.)

What functional testing does not tell you

This is the honest boundary of the technique. A fully green functional suite proves the software computes the right answers — and says nothing about:

• Speed. analyze can be correct and still take ten minutes on a large file. Functional tests don’t time anything; that’s performance testing.
• Security. A GET/SET protocol can be functionally perfect and still leak data or accept an injection. Correct behavior on expected input says nothing about malicious input; that’s security testing.
• Feel and reach. The output can be exactly right and still be unreadable, inaccessible, or broken on one browser. That’s accessibility, usability, and compatibility.
• The unknown unknowns. Functional tests check the cases you thought of. The bug you didn’t imagine won’t be in the suite — which is why exploratory testing exists to hunt for it.

Functional testing is necessary and foundational. It is not sufficient. Holding those two facts at once is the whole point of this Part.

The architect’s lens

• Why does it exist? Because “it works on my machine” is a feeling, not evidence. Functional testing turns a vague belief that the software is correct into a repeatable, falsifiable claim: given this input and state, the output is this — and here is the check that proves it.
• What problem does it solve? It closes the gap between what the software was specified to do and what it actually does, at every level from a single function to a full workflow — catching the wrong-answer bug before a user does.
• What are the trade-offs? Tests cost time to write and maintain, and they only ever check the cases you encoded. Over-specifying (asserting how instead of what) makes them brittle and breaks correct refactors; under-specifying makes them green but worthless. The craft is asserting exactly the behavior the spec promises — no more, no less.
• When should I avoid it? Never avoid it entirely — but don’t reach for a heavyweight end-to-end functional test when a fast unit-level one covers the same rule, and don’t write a functional test for a property that isn’t functional (speed, load, security) — use the technique built for that concern instead.
• What breaks if I remove it? Everything downstream loses its foundation. Regression testing has nothing to re-run; a refactor becomes a leap of faith; every other kind of testing is now measuring a program that might be computing the wrong answer. Functional tests are the ground the rest of the pyramid stands on.

Check your understanding

1. In one sentence, what does a functional test verify, and what is the one word that all of its authority depends on?
2. Why is functional testing described as a concern rather than a level? Give an example of the same functional concern expressed at the unit, integration, and end-to-end levels.
3. Black-box and white-box functional tests check the same thing but differ in one respect. What is the same, and what differs?
4. The level filter compares line.level < min. Explain why a white-box tester would immediately want a test at exactly Level::Warn, and what bug it guards against.
5. Name three things a fully passing functional suite tells you nothing about, and name the kind of testing that covers each.

Show answers

1. A functional test verifies that, given an input and a starting state, the software’s observable output/behavior matches what the specification says it should be. The load-bearing word is specification — without an agreed “should,” a test is just a guess about current behavior.
2. Because it describes what you check (correctness of behavior), which is independent of how much of the system you run (the levels). Example: unit — analyze(...) returns matched == 2 for a min WARN filter; integration — a real Db plus TCP server returns VALUE ada lovelace after a SET/GET; end-to-end — running the logwise binary prints ERROR : 3. All three check correctness; they differ only in scope.
3. Same: both assert correct observable behavior against the spec — that’s what makes both “functional.” Different: how the test cases are chosen. Black-box derives cases from the contract only, ignoring internals; white-box reads the code and picks cases to exercise specific branches and boundaries.
4. Because < makes the comparison exclusive, so the exact question “is a line at min kept or dropped?” lives right at that operator. A test with min_level = WARN asserting matched == 2 (WARN kept, plus ERROR above) guards against an off-by-one where WARN is wrongly excluded (or <= would wrongly include the level below). It pins down whether min is inclusive.
5. It tells you nothing about speed (→ performance testing), security (→ security testing), and feel/reach — usability, accessibility, browser compatibility (→ accessibility, usability & compatibility testing). It also can’t cover the cases you never imagined (→ exploratory testing).