Performance Testing — Load, Stress, Spike & Soak
Every kind of testing you have met so far in this part — functional, regression, smoke and sanity — asks one core question: does the system produce the right answer? Give it an input, check the output, compare against what the spec says it should be. Correctness is a property of a single request in isolation.
Performance testing asks a completely different question, and it is the reason it belongs to the non-functional family. A checkout endpoint can be 100% correct — right total, right tax, right receipt — and still be useless, because it takes eleven seconds to respond under Black Friday load, or because it leaks a little memory on every request and falls over at 3 AM after running fine all day. Nothing was wrong with any single answer. The system failed at scale, over time, under pressure. That is the axis this page attacks: not “is it correct?” but “is it fast and stable enough under the real and extreme conditions it will actually meet?”
The shared method of the performance family
Section titled “The shared method of the performance family”Load, stress, spike, and soak are not four unrelated tools. They are four shapes of the same experiment, and the experiment is always the same three steps:
- Drive traffic — generate requests (or jobs, or events) against a running system at a chosen rate and pattern.
- Measure — record latency (how long each request took), throughput (how many per second the system completed), and resource use (CPU, memory, connections, disk, queue depth).
- Compare against a target — decide pass or fail by holding those measurements against an explicit goal: a Service Level Objective (SLO), a budget, a breaking-point you wanted to locate.
┌─────────────┐ requests ┌──────────────┐ │ load │ ──── at rate R ───► │ system │ │ generator │ │ under test │ └─────────────┘ └──────┬───────┘ ▲ │ │ measures ◄────────────────────────┘ │ • latency (p50, p95, p99) │ • throughput (requests/sec completed) │ • resources (CPU %, memory, connections) │ ▼ compare to TARGET ──► pass / fail (e.g. "p99 < 300ms at 2,000 rps, memory flat over 4h")What changes between the four is only the traffic pattern you drive and how long you drive it — the flat, the escalating, the sudden, and the sustained. Everything else — measure, compare to a target — is identical. Learn the method once; the four techniques are just four inputs to it.
LOAD STRESS SPIKE SOAK expected/peak push past the sudden surge hold steady normal traffic limit then drop for hours/days
rps rps rps rps │ ▁▄██▄▁ │ ▁▄████ │ █ │ ████████████ │ ▁████████▁ │ ▁▄███████ │ █ │ ████████████ │▁██████████▁ │▁▄██████████ │▁▁▁▁█▁▁▁▁ │ ████████████ └──────────► t └──────────► t └──────────► t └──────────────► t "holds at scale?" "where/how break?" "survive the "stable over surge?" time?"Load testing — does it hold at realistic scale?
Section titled “Load testing — does it hold at realistic scale?”Load testing drives the traffic you actually expect — normal load and peak-normal load — and asks whether your performance targets still hold there. It is not trying to break anything. It is trying to establish confidence that the SLOs you promised are true at the scale you will really run.
The critical inputs are honesty about two numbers: your expected steady traffic and your peak traffic (the busiest realistic minute — the lunchtime rush, the Monday-morning login storm, the last hour before a deadline). You drive both and check that latency and error rate stay inside their SLOs at each.
An SLO makes the pass/fail line explicit before you run, so the test has a verdict instead of a vibe. A typical one:
SLO (the target the load test checks): "99% of GET /checkout requests complete in under 300ms, with an error rate below 0.1%, at a sustained 2,000 rps."
Load test result: throughput held at 2,000 rps ✓ p99 latency = 271ms (< 300ms) ✓ error rate = 0.04% (< 0.1%) ✓ → PASS: the SLO holds at peak-normal scaleWithout a load test, “we think it’ll handle Black Friday” is a hope. With one, it is a measured claim with a number attached.
Stress testing — where and how does it break?
Section titled “Stress testing — where and how does it break?”Load testing stays inside the expected envelope. Stress testing deliberately pushes past it — ramping traffic up and up, beyond peak, beyond what you think the system can take — for two reasons.
First, to find the breaking point: the throughput at which the system stops meeting its SLO. Knowing “we degrade past 4,500 rps” turns capacity from a guess into a number you can plan headroom against.
Second, and more importantly, to see how it fails — because every system fails eventually, and the only choice you get is graceful versus catastrophic:
GRACEFUL degradation CATASTROPHIC failure ──────────────────── ──────────────────── past the limit, the system: past the limit, the system: • sheds load (rejects excess • accepts everything, queues with a fast 503 + Retry-After) unboundedly, runs out of memory • protects the core with a • one slow dependency stalls all circuit breaker / bulkhead threads → total lockup (cascade) • serves stale-but-valid cache • crashes, restarts into the same • latency rises, but it STAYS UP flood, crashes again (crash loop)
recovers when load drops ✓ needs a human at 3 AM ✗A stress test’s real payoff is not the breaking-point number — it is discovering which of these two behaviors you have while it is still cheap to fix. A system that returns a fast, honest “try again in 5 seconds” at 6,000 rps is healthy. A system that silently accepts all 6,000, buffers them into oblivion, and takes down every other service that depends on it is a landmine — and you want to step on it in a test, not in production.
Spike and soak — the surge and the marathon
Section titled “Spike and soak — the surge and the marathon”Load and stress vary how much traffic. Spike and soak vary the shape over time, and each surfaces a class of bug the others cannot.
Spike testing — the sudden surge
Section titled “Spike testing — the sudden surge”A spike test jumps traffic from near-idle to very high almost instantly, then (usually) drops it again — modelling the real world’s discontinuities: a link goes viral, a TV ad airs, a flash sale opens, a dependent service recovers and dumps its entire backed-up queue on you at once.
The failure spike testing catches is not about the steady high rate (load testing covers that) — it is about the transient. Auto-scaling that takes 90 seconds to add capacity is useless against a surge that arrives in 2 seconds; connection pools that were sized for the average get exhausted in the first instant; cold caches all miss at once and hammer the database. A system that is perfectly happy at a ramped 3,000 rps can fall over when the same 3,000 arrives as a wall. Spike testing also checks recovery: after the surge passes, does the system return to normal, or is it wedged?
Soak testing — the slow marathon
Section titled “Soak testing — the slow marathon”A soak test (also called endurance or longevity testing) does the opposite: a modest, sustained load held for a long time — hours, or days. The traffic is unremarkable. Time is the variable.
Soak testing exists to catch the bugs that only appear with accumulation — the ones invisible in a ten-minute run because they need to build up:
- Memory leaks — a few kilobytes not freed per request is nothing at minute one and an out-of-memory crash at hour twenty.
- Resource exhaustion — file descriptors, database connections, or threads leaked slowly until the pool is empty and every new request hangs.
- Slow degradation — a cache or table that grows unbounded, a log that fills the disk, fragmentation that makes each hour a little slower than the last.
A soak test's tell-tale chart — latency FLAT, memory CREEPING:
memory │ ╱ OOM crash │ ╱───── │ ╱───── ← the leak: a slow, │ ╱───── steady climb no │ ╱───── short test would see │ ╱───── │╱───── └────────────────────────────────────► time (hours) 10min run stops here ▲ — looks perfectly healthy!The whole point of a soak test is that everything looks fine for the first ten minutes — which is exactly why a short load test gives it a clean bill of health right up until it dies overnight.
The metrics that make it honest
Section titled “The metrics that make it honest”The measurements are where performance testing is most often done wrong. Three principles separate a real performance test from a misleading one.
Percentiles, not averages
Section titled “Percentiles, not averages”Never trust an average latency. An average hides its worst cases, and the worst cases are exactly what hurt users. Report percentiles: p50 (median — a typical request), p95, p99 (the slowest 1%). The p99 is not an edge case you can ignore — on a page that makes 20 backend calls, the odds that at least one hits the p99 are high, so your p99 becomes many users’ typical experience.
1,000 requests. 990 take 50ms, 10 take 5,000ms (a stalled dependency).
average = (990·50 + 10·5000) / 1000 = 99.5ms ← "looks fine!" p50 = 50ms p99 = 50ms p99.9 = ~5,000ms ← the truth: a real stall, hidden by the mean and even p99The average and the mean whisper “99ms, all good.” The tail percentile screams “one in a thousand users waits five seconds.” Only the tail told the truth.
Realistic workloads
Section titled “Realistic workloads”A load generator hammering one cached endpoint with identical requests measures almost nothing — it tests your cache, not your system. Real traffic is a mix: reads and writes, hot and cold keys, big and small payloads, logged-in and anonymous, with think-time between a user’s actions. An unrealistic workload gives a beautiful, meaningless number. Model the distribution of what real users do, or you are load-testing a fantasy.
A production-like environment
Section titled “A production-like environment”Numbers from a scaled-down staging box do not transfer. Half the RAM, a smaller connection pool, an empty database instead of one with 200 million rows, no noisy neighbours — every one of these can flip a pass into a fail. Measure in an environment that matches production in the dimensions that affect performance (data volume, instance size, network topology), or state loudly that your numbers are directional only.
Under the hood — coordinated omission, the lie in your own tool
Section titled “Under the hood — coordinated omission, the lie in your own tool”A subtle trap can make your load tool under-report latency by an order of magnitude. Many naive generators work in a loop: send a request, wait for the response, then send the next. If the system stalls for 2 seconds, that thread simply sends fewer requests during the stall — the requests that would have been sent during the freeze are never counted. The tool omits exactly the slow samples, in a way coordinated with the stall. This is coordinated omission (a term popularised by Gil Tene). Your histogram looks great precisely because the system was slow.
Intended: fire a request every 10ms. At t=1000ms the server freezes for 2000ms.
naive tool: sends at 1000ms, blocks until 3000ms, records ONE 2000ms sample — and never records the ~200 requests that should have fired during the freeze at all. honest tool: knows those 200 requests were DUE; back-fills them with their true (rising) latency → p99 tells the truth.The fix is a generator that measures latency against when a request was scheduled to be sent, not when it happened to be sent — or one that drives an open-model, fixed-rate load. Without it, the very freezes you are hunting are the ones your tool hides. Verify your load tool corrects for coordinated omission before you trust a single percentile it prints.
The architect’s lens
Section titled “The architect’s lens”- Why does it exist? Because correctness is not sufficiency. A system can compute every answer right and still fail its users by being too slow at peak, collapsing under a surge, or leaking until it dies overnight — failures no functional test can see, because they are properties of scale, time, and pressure, not of any single request.
- What problem does it solve? It converts “we think it’ll hold” into a measured claim with a number and a verdict: the SLO holds at 2,000 rps (load), we break at 4,500 rps and shed load gracefully (stress), we survive a 10× surge and recover (spike), memory is flat over 48 hours (soak). It finds the slow, the fragile, and the leaky before a user, a sale, or a night does.
- What are the trade-offs? It is expensive to do well: a production-like environment, a realistic workload model, load-generation infrastructure, and long-running soak windows all cost real time and money. Done cheaply — averages instead of percentiles, one endpoint instead of a mix, a tiny staging box — it produces confident numbers that are simply wrong, which is worse than no test at all.
- When should I avoid it? When the stakes do not justify the setup: a low-traffic internal tool, an early prototype still changing shape daily, or a batch job with no latency requirement rarely needs a full performance harness. Reach for it when scale, spikes, or long uptime are real risks — not reflexively on every change.
- What breaks if I remove it? Nothing — until load, a surge, or time arrives. Then the failures ship silently: the checkout that is correct but eleven seconds slow at peak, the service that cascades into total lockup under a spike, the memory leak that crashes production at 3 AM after passing every functional test all day. Performance testing is the only family that catches “correct but unusable,” and without it those defects reach users at the worst possible moment.
Check your understanding
Section titled “Check your understanding”- Performance testing belongs to the non-functional family. In one sentence, how is the question it asks fundamentally different from the question a functional test asks — and give an example of a system that would pass one and fail the other.
- Load, stress, spike, and soak share one three-step method. State the three steps, and explain what varies between the four techniques (and what stays the same).
- A stress test’s most valuable output is often not the breaking-point number. What is it, and why does “graceful vs catastrophic” matter more than “how many rps”?
- Why can a system that is perfectly stable under a 10-minute load test still die overnight in production — and which technique is designed to catch that, driving on which variable?
- Explain why you should report p99 latency rather than average, and why the p99 of a single backend service can become the typical experience of a user on a page with heavy fan-out.
Show answers
- A functional test asks “is the answer correct?” (a property of one request in isolation); performance testing asks “is it fast and stable enough under real and extreme conditions?” (a property of scale, time, and pressure). Example: a checkout endpoint that returns the exactly-correct total but takes 11 seconds under peak load passes every functional test and fails a load test — correct, yet unusable.
- The method: (1) drive traffic at a chosen rate/pattern, (2) measure latency, throughput, and resource use, (3) compare against an explicit target (an SLO, budget, or breaking-point) for a pass/fail verdict. What varies between the four is only the traffic pattern and duration you drive — flat expected/peak (load), escalating past the limit (stress), a sudden surge (spike), a modest load held for a long time (soak); the measure-and-compare steps are identical.
- The more valuable output is how the system fails — graceful (sheds excess load with fast 503s, circuit-breaks, serves stale cache, stays up and recovers) versus catastrophic (queues unboundedly, runs out of memory, cascades into total lockup, crash-loops). It matters more because every system breaks eventually, so the only real choice is the failure mode; a system that degrades gracefully survives a surprise, one that fails catastrophically takes everything down and needs a human at 3 AM.
- Because some defects only appear with accumulation over time — memory leaks, slowly-exhausted file descriptors or connection pools, unbounded cache/log/disk growth — all of which are invisible in a 10-minute run and only crash the system after hours or days. Soak (endurance) testing catches these: it holds a modest, sustained load for a long time, driving on the variable of time rather than traffic volume.
- An average hides its worst cases by blending them into the mean, so a real stall (a few requests at 5,000ms) can vanish behind a healthy-looking 99ms average — while the p99/p99.9 tail exposes the slowest requests, which are exactly what hurt users. On a page with heavy fan-out (say 20 backend calls that must all return), the probability that at least one call hits its p99 tail is high, so the per-service “rare” slow case becomes the per-user common case — tail amplification — which is why the tail, not the average, is what you optimise.