Strings — Empty, Huge, Unicode, and Injection

Numbers had a short, knowable list of landmarks — zero, one, MIN, MAX, the range edges. Strings feel similar at first: surely there’s an empty one, a normal one, and a big one, and you’re done. That intuition is what makes text the richest source of bugs in this book. A string is not a tidy value on a number line; it is a variable-length sequence of bytes that pretends to be a sequence of characters, and almost every assumption you make about that pretence is false at some input.

This page teaches you to attack text on purpose. We walk from the degenerate strings (empty, whitespace, stray spaces) through the lie that a string has a single obvious “length,” into encoding and normalization traps that make two identical-looking strings compare unequal, and finally to the edge case with a security budget attached: injection, where attacker text stops being data and becomes code. Every section is a class of input you must enumerate and pin down with a hostile test — not a friendly one.

The degenerate strings come first

Before any interesting Unicode, there is a small set of strings that break naive code precisely because they look like nothing. Enumerate these at every text input, the same way you enumerate zero and one for a number:

""              the empty string          (length 0, still a valid string)
" "             a single space            (non-empty, but "blank")
"   \t\n  "     whitespace-only           (looks empty, isn't)
"  hi"          leading whitespace
"hi  "          trailing whitespace
" hi "          both
" "        a non-breaking space       (whitespace your regex may miss)

The empty string is the zero of text. It is a valid string of length 0, and it slips through every check that asks “is this null?” while failing every check that assumes “if it exists, it has content.” A required username field that rejects null but accepts "" has a hole. The test is the boring one everyone skips:

def test_username_rejects_empty_and_blank():
    assert not is_valid_username("")        # the empty string
    assert not is_valid_username("   ")     # whitespace-only
    assert not is_valid_username("\t\n")    # other whitespace
    assert is_valid_username("alice")       # a real one, for contrast

Trim order changes the answer

The subtle bug here is not whether you trim, but when. Consider validation that checks length, then trims for storage:

Input:  "   "   (three spaces)

Order A — validate then trim:
  len("   ") == 3   →  passes "must be non-empty"
  trim → ""         →  an empty username is now stored.   BUG.

Order B — trim then validate:
  trim("   ") → ""
  len("") == 0      →  fails "must be non-empty".         CORRECT.

Same two operations, opposite outcomes, decided purely by order. The rule is normalize first, validate second — trim, case-fold, and normalize the input before you judge it, so you validate the value you will actually store. A test that only feeds "alice" never sees this; a test that feeds " " and " alice " does.

Length is a lie

Ask “how long is this string?” and the honest answer is “in which unit?” A string has at least three different lengths, and the one your language’s .length returns is usually not the one a human means.

"a"          bytes: 1   code points: 1   graphemes: 1    (all agree — the trap)
"café"       bytes: 5   code points: 4   graphemes: 4    (é is 2 UTF-8 bytes)
"👍"          bytes: 4   code points: 1   graphemes: 1    (one emoji, 4 bytes)
"👨‍👩‍👧"        bytes: 18  code points: 5   graphemes: 1    (a family: 3 people + 2 joiners)
"é" (NFD)    bytes: 3   code points: 2   graphemes: 1    (e + combining accent)

Three distinct notions of “length”:

Bytes — how much storage the string occupies once encoded. This is what a VARCHAR(20) database column, a network buffer, or a byte-limited field actually counts.
Code points — the Unicode scalar values. This is what Rust’s .chars().count() and Python 3’s len() return.
Grapheme clusters — what a human perceives as one character: 👨‍👩‍👧 is one glyph but five code points glued by zero-width joiners. This is what “delete one character” or “the string is too long for the UI” should mean.

Your language picks a default and hides the other two. JavaScript’s .length counts UTF-16 code units, so "👍".length is 2, not 1, and a single astral-plane character can report a length of 2 — surprising every developer who assumed one visible character equals one unit. Rust’s String indexes by bytes and will refuse to compile s[0]; you must choose .bytes(), .chars(), or a grapheme crate deliberately.

Naive slicing splits a character in half

The real damage from the length lie is truncation. “Trim this to 100 characters for the preview” is trivial until the 100th boundary lands in the middle of a multi-byte character or a grapheme cluster.

Truncate "café" to 4 bytes:
  UTF-8 bytes:  63 61 66 C3 A9        ("é" is the two bytes C3 A9)
  take 4:       63 61 66 C3           ← cut between C3 and A9
  result:       "caf" + a lone C3     → invalid UTF-8 / a replacement char �

Truncate "👍" to 2 bytes:
  bytes:  F0 9F 91 8D
  take 2: F0 9F                        → half an emoji, not a character

Cut a grapheme cluster and you get worse: slicing "👨‍👩‍👧" after two code points yields a man plus a dangling joiner, rendered as either a lone man or a broken box. The lesson is that length limits must be expressed in the unit you claim to mean, and truncation must land on a legal boundary. Test with a string whose Nth boundary falls inside a multi-byte character; a pure-ASCII test can never expose it.

// Rust makes the trap a compile-and-runtime check, which is a gift:
let s = String::from("café");
// let bad = &s[0..4];   // panics at runtime: not a char boundary
let ok: String = s.chars().take(3).collect();  // "caf" — safe by construction
assert_eq!(ok, "caf");

Encoding boundaries and normalization

A string in memory is not text until you agree on an encoding — the rule mapping bytes to characters. Get the rule wrong and the same bytes become different text, or no text at all.

UTF-8    variable width, 1–4 bytes/char, ASCII-compatible. The web's default.
UTF-16   2 or 4 bytes/char, surrogate pairs for the astral planes. Java/JS/Windows internals.
Latin-1  one byte per char, only 256 characters. Legacy, and a frequent mojibake source.

Two failures live at this boundary. First, invalid byte sequences: not every byte string is valid UTF-8. A lone continuation byte, an overlong encoding, or an unpaired UTF-16 surrogate is malformed input. Your parser must decide — reject, or replace with � (U+FFFD) — and you must test that decision with deliberately malformed bytes, because attackers send them on purpose (the overlong-UTF-8 path-traversal tricks of the early 2000s exploited exactly this).

Normalization: equal-looking, not equal

Unicode often offers more than one way to encode the same visible character. “é” can be a single code point (U+00E9, NFC, “composed”) or an “e” followed by a combining acute accent (U+0065 U+0301, NFD, “decomposed”). They render identically. They are not byte-equal, so == says they differ.

"é" as NFC:  U+00E9              bytes: C3 A9        len (code points) = 1
"é" as NFD:  U+0065 U+0301       bytes: 65 CC 81     len (code points) = 2

  nfc == nfd            →  FALSE   (different bytes, different length)
  normalize(nfc, NFC) == normalize(nfd, NFC)   →  TRUE

This is why a user “can’t log in with the right password,” why a search misses an obviously-present record, and why a “duplicate” row sneaks past a uniqueness check. The fix is to normalize to a single canonical form (typically NFC) at every trust boundary before comparing, hashing, indexing, or storing — the same “normalize first” rule from the whitespace section, now for accents. The test feeds the two encodings of the same word and asserts they are treated as one.

The string  "e" + U+0301 + "👨‍👩‍👧"   ("é" decomposed, then a family emoji)

  UTF-8 bytes         3 + 18           = 21 bytes
  UTF-16 code units   2 + 8            = 10 units   → JS ".length"
  Unicode code points 2 + 5            = 7          → Python len()
  grapheme clusters   1 + 1            = 2          → what a human counts

One string, four different "lengths": 21, 10, 7, 2. A VARCHAR(8) column
(8 *characters*, if it means code points) rejects this two-glyph string —
and a UI that budgets "20 characters" by bytes truncates it mid-family.

Very long and adversarial strings

The empty string is one edge; the pathologically long one is the other. Text has no natural MAX the way an i32 does, so you must impose one — and test both the limit and the monster just past it.

""                  the empty edge
"a" × 1             one character
"a" × (limit)       exactly at the declared maximum
"a" × (limit + 1)   one past — must be rejected, not truncated or crashed
"a" × 10_000_000    a hostile blob — does it OOM, hang, or bound cleanly?

An unbounded text field is a denial-of-service invitation: a 100 MB “name” can exhaust memory, blow a buffer, or wedge a database. Enforce a maximum length, and test the value one past it.

ReDoS: a regex that hangs on a crafted string

Some inputs are adversarial not by size but by shape. A regex with nested or overlapping quantifiers — (a+)+$, (a|a)* — can take exponential time on a non-matching string, because the engine backtracks through every way to split the input. This is ReDoS (regular-expression denial of service): a few dozen characters freeze a CPU for seconds or minutes.

Pattern:  ^(a+)+$
Input:    "aaaaaaaaaaaaaaaaaaaaaaaaX"   (24 a's, then a non-match)

  matching attempts grow ~2^n as n rises.
  24 a's → millions of backtracks → seconds of 100% CPU on one request.

The lesson is threefold: prefer linear-time regex engines (RE2, Rust’s regex crate) that forbid catastrophic backtracking; keep patterns simple and anchored; and add a fuzz test that throws long, repetitive, near-match strings at every regex and asserts it returns within a time budget.

Embedded null bytes

C-family code treats the null byte \0 as “string ends here”; higher-level languages treat it as an ordinary character. That mismatch is an exploit. "safe.txt\0.php" looks like it ends in .php to a length-aware validator and reads as safe.txt to a null-terminated filesystem call — bypassing the extension check entirely. Test every input that crosses a language boundary with an embedded \0 and assert it is rejected or that the null survives intact end to end.

Injection: the edge case with a CVE

Every injection bug — SQL, shell command, HTML/XSS, LDAP, log — is the same mistake wearing different clothes: attacker-controlled text was pasted into a structured string and then interpreted as structure instead of data. The user’s input stopped being a value and became syntax.

Intended query, with name = "alice":
  SELECT * FROM users WHERE name = 'alice'

Attacker sends name = ' OR '1'='1
  SELECT * FROM users WHERE name = '' OR '1'='1'
                                        └──────── always true → dumps every row

Attacker sends name = '; DROP TABLE users; --
  SELECT * FROM users WHERE name = ''; DROP TABLE users; --'
                                       └──────── a second statement

The root cause is string concatenation building code. The cure is to stop concatenating and instead keep data and structure in separate channels:

SQL — use parameterized queries / prepared statements. The ? placeholder is sent to the database as structure; the value travels separately and can never become syntax. Never build SQL with string formatting.
Shell — never pass a built string to a shell. Use exec with an argument array (["ls", user_input]), so the input is one argument, not a fragment of a command line the shell re-parses.
HTML — escape on output (< → <) so text renders as text, or use a templating engine that auto-escapes, plus a Content-Security-Policy as defense in depth.

# WRONG — concatenation lets data become code:
cur.execute("SELECT * FROM users WHERE name = '" + name + "'")

# RIGHT — the value can never be parsed as SQL:
cur.execute("SELECT * FROM users WHERE name = ?", (name,))

Injection is an edge case because it lives at the boundary between two languages (your language and SQL/shell/HTML), and the defect is invisible to any test that only sends polite input. You must test with hostile payloads — ' OR '1'='1, '; DROP TABLE, <script>alert(1)</script>, $(rm -rf /), a null byte, a Unicode look-alike — and assert the input is treated as an inert value, never executed. A test suite of well-behaved names proves the happy path and nothing about safety.

For most of the 2000s and 2010s, injection sat at or near the top of the OWASP Top 10 list of web-application risks, and SQL injection was implicated in a long series of large, dated breaches — from the 2008 Heartland Payment Systems intrusion (tens of millions of card records) to the 2011 Sony and 2015 TalkTalk incidents, each reportedly reached in part through injectable inputs.

Every one traces to the same root: user text concatenated into a query or command and then interpreted as structure. The permanent lesson is not a specific patch but a discipline — parameterize and escape at every trust boundary, and prove it with hostile-payload tests, not with the friendly inputs that pass regardless. Treat the exact figures as approximate; the enduring point is the mechanism.

The architect’s lens

Why does it exist? Because a string only pretends to be a sequence of characters — it is really variable-length bytes under an encoding — and every convenient assumption (one length, == means “same text,” input is inert data) is false at some input.
What problem does it solve? It replaces “it worked on alice” with explicit checks at the empty string, whitespace, multi-byte and grapheme boundaries, both normalization forms, the length limit, adversarial regex inputs, and hostile injection payloads — the places where text actually breaks.
What are the trade-offs? Doing text right means knowing your language’s default length unit and encoding, normalizing at every boundary, and maintaining hostile-input fixtures. That knowledge and vigilance are the cost; the alternative is a corruption or a breach.
When should I avoid it? Never skip the degenerate strings or the injection payloads — those are cheap and catch the worst bugs. You can defer deep grapheme-cluster handling for internal, ASCII-only fields, but only after deciding so on purpose, not by forgetting.
What breaks if I remove it? You ship code that accepts a blank username, truncates an emoji into �, rejects a valid login because the accent was decomposed, hangs on a crafted regex, and hands an attacker a DROP TABLE — data becoming code, the whole point of an edge.

Check your understanding

Name three “degenerate” strings you would test at every text input, and explain how validating before trimming can let an empty value slip through while trimming before validating catches it.
A string has at least three different “lengths.” Name them, say which one JavaScript’s .length and Rust’s byte indexing use, and explain why "👍".length surprises people.
Why does naive truncation to a byte count corrupt "café" or "👍", and what does it mean to truncate “on a legal boundary”?
Two strings render as an identical “é” but compare unequal with ==. What is happening, what is the general term, and what single step fixes it before comparing or storing?
Injection (SQL, shell, HTML) is described as “one mistake in different clothes.” State the shared root cause, and give the specific defense for SQL and for the shell. Why must you test with hostile payloads rather than polite input?

Show answers

Any three of: the empty string "", a single space " ", whitespace-only "\t\n", and leading/trailing spaces like " hi " (plus non-breaking spaces). If you validate then trim, " " has length 3 and passes a “non-empty” check, but trimming later stores "" — a bug. If you trim then validate, " " becomes "" and fails the check. The rule is normalize first, validate second, so you judge the value you actually store.
Bytes (storage after encoding, what a VARCHAR/buffer counts), code points (Unicode scalars, Python 3’s len(), Rust’s .chars().count()), and grapheme clusters (what a human sees as one character). JavaScript’s .length counts UTF-16 code units and Rust indexes String by bytes. "👍".length is 2 in JS because the emoji is one code point but two UTF-16 code units, not one visible character.
Multi-byte characters occupy several bytes ("é" is C3 A9; "👍" is four bytes), so cutting at an arbitrary byte count can fall inside a character, leaving invalid UTF-8 that renders as � or half an emoji. Truncating “on a legal boundary” means cutting only between whole characters/grapheme clusters, and expressing the limit in the unit you actually mean (code points or graphemes), never mid-character.
The same visible “é” has two encodings: NFC (one composed code point U+00E9) and NFD (an “e” plus a combining accent, two code points). Different bytes and lengths make == report them as different — this is a normalization problem. The fix is to normalize both to a single canonical form (typically NFC) at the trust boundary before comparing, hashing, indexing, or storing.
The shared root cause is attacker text concatenated into a structured string and then interpreted as structure (code) rather than data. For SQL, use parameterized queries / prepared statements so the value travels in a separate channel from the query structure; for the shell, use exec with an argument array instead of passing a built string to a shell to re-parse. Polite input exercises the happy path regardless of the bug; only hostile payloads (' OR '1'='1, '; DROP TABLE, <script>, $(...), a null byte) prove the input is treated as inert data.