Resources, Network, State, and Trust Boundaries

The concurrency page attacked the schedule — the order in which independent threads interleave. This page attacks something that feels even more unfair: the environment. Your code is correct, your inputs are valid, your threads are synchronized — and the machine still runs out of memory, the network drops the connection halfway through a write, the disk fills, or a hostile client sends bytes designed to break you.

Every edge so far lived inside the program. This one lives outside it, in the physical and adversarial world the program runs in. The mental shift for this page: “it works on my machine” is a boundary, not a guarantee. Your laptop has spare RAM, a fast local disk, a network that never blinks, and no attacker. Production has none of those luxuries. The edges here are the ones that only appear when the world stops cooperating — which, at scale, is constantly.

Resource limits: memory, disk, and file descriptors

Every resource is finite. On your machine the ceiling is so high you never touch it; in production, under load, you touch all of them. Three ceilings matter most.

Out of memory (OOM). A request that allocates a buffer proportional to its input is fine for a 2 KB payload and fatal for a 2 GB one. The strings page warned about the huge input; here is where it bites. When allocation fails, behavior forks by language: a managed runtime may throw and unwind (recoverable, if you catch it); a native process may get killed outright by the kernel’s OOM killer with no chance to clean up. The test that matters is not “does it work with normal input” — it is “what happens at the size where memory runs out, and does the failure leave the system in a sane state.”

Out of disk. Disk fills silently until a write fails at the worst possible moment — often mid-write, leaving a half-written file. The repo’s kvlite server appends every SET to a write-ahead log on disk:

// rust/kvlite/tests/server.rs — every SET is durably appended to a WAL file.
let (db, _) = Db::open(&path).unwrap();
// ...
assert_eq!(round_trip(&mut reader, &mut writer, "SET name ada lovelace"), "OK");

That OK means “written to disk.” The edge case the happy test never triggers: what does SET return when the disk is full? If the append fails but the in-memory map already updated, the store now lies — it reports a value that will vanish on restart. Testing this means injecting a full disk (a tiny tmpfs, a quota, a mocked writer that returns ENOSPC), because you cannot fill a real disk on demand in CI.

File-descriptor exhaustion. Every open socket, file, and pipe consumes a descriptor, and the OS caps them (often ~1024 per process by default). A connection leak — a socket you open per request but never close — works flawlessly for the first thousand requests, then every new connection fails with “too many open files.” The kvlite server opens a fresh descriptor per client connection; a test that opens and closes ten thousand connections in a loop is exactly how you catch a leak that a two-connection test never could.

   "fits on my machine"                    the real boundary
   ────────────────────                    ─────────────────
   16 GB RAM, idle                    →    2 GB container, under load
   fast NVMe, empty                   →    disk 98% full, mid-write
   fd limit never approached          →    1024 fds, connection leak
   → all green                             → OOM kill, ENOSPC, EMFILE

Network and I/O: “failed” is not the worst outcome

On your machine the network is a function call: instant, reliable, in-process. Across a real network it is a conversation over an unreliable channel that can fail in ways a local call never can. The taxonomy that matters:

Timeout. The other side never answers. Without an explicit timeout your code waits forever, holding a thread and a connection hostage. Every network call needs a deadline; a call with no timeout is a latent hang.
Slow / partial response. The bytes arrive, but slowly, or the stream stops halfway. Your parser must handle “I have half a message and no more is coming” without hanging or crashing.
Connection drop mid-write. You sent half the request, or the server processed it and the connection died before the reply reached you. The kvlite round_trip helper writes a line and reads a reply — but on a real network the connection can die after the write and before the read.

That last case is the deep one, and it deserves its own name.

Under the hood — “failed” versus “unknown outcome”

There are not two outcomes to a network call (success, failure). There are three: success, definite failure, and unknown.

   you send: "charge $80"

   (a) reply "charged"        → success. done.
   (b) reply "declined"       → definite failure. safe to report.
   (c) ...silence... timeout  → UNKNOWN. did the charge happen or not?

Case (c) is the hard one. A timeout does not mean the operation failed — it means you don’t know. The request may have been received, processed, and committed, with only the reply lost on the way back. If you treat “unknown” as “failed” and retry, you may run the operation twice. If you treat it as “succeeded,” you may lose it. The entire discipline of safe retries exists to make “unknown” survivable — because on a real network, “unknown” is not rare; it is a routine outcome you must design for.

Retries and idempotency: making “unknown” safe

The instinct when a call returns “unknown” is to retry. That instinct is right, but a naive retry is dangerous: if the first attempt did succeed, the retry runs the operation a second time. Retry “charge the customer” and you double-charge them.

The fix is idempotency: design the operation so that doing it twice has the same effect as doing it once. The concurrency page introduced this; here it is the load-bearing tool. The standard mechanism is an idempotency key — a unique token the client generates once and attaches to every attempt of the same logical request:

   client generates key once:  "idem-key: 9f3a-..."

   attempt 1 ──charge $80, key 9f3a──►  server: key unseen → charge, record key → "charged"
                                        (reply lost on the way back — client sees timeout)
   attempt 2 ──charge $80, key 9f3a──►  server: key ALREADY seen → do NOT charge again,
                                        return the SAME result → "charged"

   result: exactly one charge, even though the client sent the request twice.

The server stores the key with the result. A duplicate key returns the stored result without re-running the effect. The client can now retry “unknown” outcomes freely — the duplicate is harmless because the server dedupes it. This is why set(key, value) is safe to retry (idempotent) but append or increment is not (each repeat changes the result).

Exponential backoff and retry storms. How you retry matters as much as whether you can. If a service is briefly overloaded and every client retries immediately and in lockstep, the retries pile onto the struggling service and keep it down — a self-inflicted retry storm that turns a blip into an outage. The fix is exponential backoff with jitter: wait longer after each failure (1s, 2s, 4s, 8s…) and add randomness so clients don’t synchronize.

   BAD  (fixed, synchronized):    retry, retry, retry — all clients at once
                                  → the overloaded service never recovers

   GOOD (exponential + jitter):   wait 1s±r, then 2s±r, then 4s±r, give up after N
                                  → load spreads out; the service gets room to breathe

Retries without a cap are also a trap — retry forever and a permanently-down dependency ties up your threads instead of failing fast. Bound the attempts, then surface the failure.

State and sequencing edges: first run, resume, double-submit

The collections page taught 0-1-many for values. The same ladder applies to state over time — the edges are the states that only happen once or out of order, so the happy path never visits them.

First run — no data yet. The very first execution has an empty database, no config file, no prior state, no cache. Code that assumes “load the last checkpoint” or “read yesterday’s total” crashes on a null it will never see again after day one. First run is the empty-list edge of state.
Resume after a crash. The process died mid-operation and restarts. Was the half-written file left behind? Is the WAL replayed correctly to reconstruct state, or does a corrupt final record halt startup? The kvlite server’s WAL exists precisely so a restart can rebuild the store — which means “kill it mid-write and restart” is a required test, not an optional one.
Double-submit — the impatient user. A user clicks “Pay” and the response is slow, so they click again. Now two identical requests are in flight. This is not a rare adversarial case; it is the single most common real-world duplicate, and it is the same problem as the network retry — an unknown outcome that produces a second attempt. The same fix works: an idempotency key on the submit, or a unique constraint the database enforces, so the second click is a harmless no-op instead of a second order.

   time ──►
   click 1:  POST /pay  ─────(slow)─────►  order created
   click 2:       POST /pay  ──────────►  same idempotency key → no-op, same result

   without a key: two orders, one angry customer.

Double-submit is the everyday face of the “unknown outcome” problem: the user, like a retrying client, cannot tell whether their action succeeded, so they repeat it. Idempotency is what makes that repetition safe.

Trust boundaries: treat every external input as hostile

A trust boundary is the line between code you control and input you don’t: the network socket, the uploaded file, the query parameter, the message off the queue. Inside the boundary you can reason about your own invariants. Crossing it, you can assume nothing — the input may be malformed, oversized, or crafted specifically to break you.

The strings page covered injection; this generalizes it into a principle: validate at the edge, and treat every external input as hostile until proven otherwise. The kvlite server does exactly this — a garbage command does not crash the server, it returns a structured error:

// rust/kvlite/tests/server.rs — malformed input is rejected at the edge, not trusted.
assert!(round_trip(&mut reader, &mut writer, "bogus").starts_with("ERR"));

That one assertion is a trust-boundary test: hostile input in, controlled error out, server still standing. The discipline scales up:

Validate structure and size at the edge — reject the 2 GB payload, the malformed JSON, the out-of-range field before it reaches business logic, so the blast radius is one rejected request, not a corrupted database.
Assume the client is lying. Client-side validation is a convenience for honest users, never a security control — an attacker skips your form and posts raw bytes. Every check that matters must run on the server.
Fail closed. When validation is ambiguous, deny rather than allow. An input you can’t confidently parse is an input you reject.

Fault and failure injection: testing paths you can’t trigger by hand

You cannot fill a real disk, drop a real packet mid-write, or exhaust real file descriptors on demand in a test suite. So you inject the failure. Fault injection deliberately makes a dependency misbehave — a mock that returns ENOSPC, a proxy that drops connections, a network layer that adds 30 seconds of latency — so the error path runs under test instead of only in production.

   real world (can't summon on demand)      injected (deterministic, in CI)
   ──────────────────────────────────      ───────────────────────────────
   disk fills mid-write               →     mock writer returns ENOSPC
   peer vanishes after your write     →     proxy kills socket post-write
   dependency times out               →     stub sleeps past the deadline
   10,000 connections leak            →     loop opens/closes 10k sockets

The principle is the same one that ran through the concurrency page: if you cannot hope to hit the bad case, force it. Error-handling code that has never executed is not “probably fine” — it is untested code, and untested error paths are where the worst outages hide.

A connection leak is invisible until you cross the limit. Suppose a service leaks one file descriptor per request against a default soft limit of 1024:

   default fd soft limit (typical):   1024
   descriptors leaked per request:    1
   requests before exhaustion:        ~1024

   at 50 requests/second:  1024 / 50  ≈  20 seconds to failure

The service runs perfectly for the demo, the code review, and the first twenty seconds of real traffic — then every new connection fails at once with “too many open files,” and the failure looks like a total outage with no code change to blame. A test that opens and closes far more than 1024 connections turns a 20-second production time-bomb into a red bar in CI. (Figures illustrative; limits vary by OS and configuration.)

On 28 February 2017, a large portion of Amazon S3 in the US-EAST-1 region became unavailable for several hours, taking down a wide swath of the internet that depended on it. The trigger, per Amazon’s public summary, was an operator command run during routine debugging that removed more capacity than intended; subsystems that had not been fully restarted in a long time then took far longer to recover than expected, and the resume path — bringing a large system back from cold — turned out to be a rarely-exercised, slow, and fragile edge. The lessons land on this page directly: resource and capacity limits are real ceilings, not abstractions; the recovery and resume path is a first-class edge case that must be tested, not assumed; and a system that has run happily for months can hide a catastrophic startup or scaling edge that only appears the one time you need it to come back. Specifics are summarized and may be imprecise; the lesson stands.

The architect’s lens

Attacking the environment — resources, network, state, and the trust boundary — is a major technique in the edge-case field guide. At altitude:

Why does it exist? Because “it works on my machine” is a boundary, not a guarantee: real deployments run with finite memory, disk, and descriptors, over an unreliable network, and against clients who may be hostile — conditions your development laptop almost never reproduces.
What problem does it solve? It surfaces the failures the happy path structurally cannot reach — OOM and full disks, timeouts and connection drops, double-submits, first-run and crash-resume states, and malicious input — so the error paths are tested before production triggers them for you.
What are the trade-offs? These paths must usually be injected (mocks, proxies, fault layers, tmpfs) rather than triggered naturally, which adds test infrastructure and complexity; idempotency keys, backoff, and edge validation add coordination and code you must build and maintain even for requests that never fail.
When should I avoid it? For a pure, in-process function with no I/O, no shared resource, and no external input, there is no environment to attack — spend the budget on the numeric, string, and collection edges instead. Match the effort to the actual attack surface.
What breaks if I remove it? The failures that make headlines walk straight in: silent data loss on a full disk, hangs from missing timeouts, double charges from naive retries and double-clicks, cascading retry storms, crashes on first run, and security holes from trusting input at the boundary — each one green in every happy-path test.

Check your understanding

Explain the claim “it works on my machine is a boundary, not a guarantee” using two specific resource ceilings that a development laptop hides but production exposes.
A network call returns neither success nor a definite error — it times out. Why is “unknown outcome” fundamentally different from “failed,” and why does treating a timeout as a failure and blindly retrying cause a bug?
What is an idempotency key, and walk through how it makes a duplicated “charge the customer” request safe when the first reply was lost to a timeout.
What is a retry storm, and how do exponential backoff and jitter prevent client retries from keeping an overloaded service down?
Define a trust boundary. Why is client-side validation never a security control, and why must error paths like a full disk or a dropped connection be tested with fault injection rather than by hoping to trigger them?

Show answers

Your laptop has abundant RAM, a fast and nearly-empty disk, and never approaches the file-descriptor limit, so resource-exhaustion paths never execute during development. Production runs in a small container that can hit OOM under load, on a disk that may be 98% full (so a write fails mid-operation with ENOSPC), and can exhaust its ~1024 descriptors from a connection leak. The same code that is “all green” locally hits ceilings production crosses routinely — so passing locally guarantees nothing about the boundary.
A definite failure (e.g. “declined”) tells you the operation did not happen, so reporting it is safe. A timeout tells you only that you got no reply — the operation may have been received, processed, and committed, with just the response lost on the way back. If you assume “failed” and retry a non-idempotent operation, the first attempt may have already succeeded, so the retry runs it a second time (e.g. a double charge). “Unknown” must be handled as its own case, made safe by idempotency rather than by guessing.
An idempotency key is a unique token the client generates once and attaches to every attempt of the same logical request. The server records the key with the result of the first attempt. If the reply is lost and the client retries with the same key, the server sees the key already exists, skips re-running the effect, and returns the stored result — so exactly one charge happens even though the request was sent twice. The duplicate becomes a harmless no-op.
A retry storm is when many clients, on seeing failures from a briefly overloaded service, all retry immediately and in lockstep, piling load onto the struggling service and preventing its recovery — turning a blip into an outage. Exponential backoff makes each client wait progressively longer after each failure (1s, 2s, 4s…), and jitter adds randomness so clients don’t synchronize their retries; together they spread the load out over time and give the service room to recover, and a retry cap ensures a permanently-down dependency fails fast instead of tying up threads forever.
A trust boundary is the line between code you control and input you don’t (a network socket, an uploaded file, a query parameter, a queue message); everything crossing it must be treated as hostile until validated. Client-side validation is never a security control because an attacker simply bypasses your form and sends raw bytes directly, so every check that matters must run on the server. Error paths like a full disk or a mid-write connection drop cannot be summoned on demand in a test suite, so you inject the fault (a mock writer returning ENOSPC, a proxy that kills the socket, a stub that sleeps past the deadline) to force the error path to run under test — because untested error-handling code is where the worst outages hide.