The Machine Room
0%

The Backend Ladder  ·  Deep dive  ·  Pillar A of three

The machine
room.

The overview page named the six mechanisms. This one opens them up: the laws that decide how much traffic a box can take, the arithmetic behind a cache, the write paths under an index, and the exact shapes failure takes when a system stops coping. Every section is a mechanism drawn, then the numbers that make it real.

L = λWOne capacity formulaIn-flight work equals arrival rate times latency. Almost every sizing question is this rearranged.
×5Latency at 80% loadQueueing multiplies wait time long before a resource is full. The knee, not the ceiling, is the limit.
99.6%Four 99.9% hopsAvailability multiplies downward in series. Composition is the number that matters, not the component.
Part one

Numbers and laws

Three quantitative tools do most of the work in backend design. Knowing them turns "it feels slow" into "we are at 85% utilisation on a single-threaded resource, so the wait is six times the service time".

Foundations deep · 01

The latency scale you argue from

Every design argument is really a claim about which of these numbers you are paying. They span nine orders of magnitude, so intuition fails — you have to know roughly where each operation sits.

LOGARITHMIC — EACH TICK IS 10× THE ONE BEFORE 1ns 100ns 10µs 1ms 100ms 1s L1 cache 1ns main memory 100ns NVMe read 100µs cross-AZ hop 1ms cross-region 75ms mutex lock 20ns ctx switch 1µs same-AZ hop 0.5ms DB query 10ms cold start 500ms
Two rules fall straight out of this picture. First: a network hop costs roughly ten thousand memory accesses, so removing a hop beats optimising the code inside it. Second: the gap between a cache hit and a cross-region call is five orders of magnitude — which is why a 1% miss rate can dominate your p99 even when 99% of requests are fast.
Foundations deep · 02

Little’s Law: the only capacity formula you need

Work in progress equals arrival rate times time in system: L = λW. It holds for any stable system, with no assumptions about distributions. Thread pools, connection pools, queue depths and pod counts all fall out of it.

HEALTHY — POOL OF 200, 100 IN USE λ = 2,000 rps W = 50ms in the system L = 2,000 × 0.05 = 100 in flight pool: 100 of 200 slots busy — headroom for a spike THE DEPENDENCY SLOWS DOWN — NOTHING ELSE CHANGES λ = 2,000 rps W = 150ms (DB p99 crept up) L = 2,000 × 0.15 = 300 in flight +100 with nowhere to go pool exhausted → requests queue → W rises again → L rises again A pool sized for today’s latency is a trap: it fails not when traffic grows, but when something downstream gets slower.
The feedback loop in the lower half is why outages are sudden. Latency rises → in-flight count rises → the pool saturates → new requests wait for a slot, which adds to W → in-flight rises further. Size pools from p99 latency with headroom, and put a bound plus a timeout on the wait so the loop has a brake.

Three questions Little’s Law answers immediately

How many workers? Throughput target × per-item time. 500 msg/s at 200 ms each needs 100 concurrent consumers. How deep is my queue allowed to be? Depth ÷ drain rate = added latency; a 10,000-item queue draining at 500/s is 20 seconds of delay, so if your SLO is 2s the bound is 1,000. Why did connections run out? Because W grew, not because λ did.

Foundations deep · 03

Why systems fall over at 80%, not 100%

Waiting time does not grow linearly with load. For a single queued resource the wait multiplier is 1/(1−ρ), which is flat and forgiving until it is neither. Capacity planning is really about staying left of the knee.

1× 5× 10× 15× 20× 0% 50% 75% 98% UTILISATION ρ — how busy the resource already is RESPONSE TIME, AS A MULTIPLE OF SERVICE TIME THE KNEE — ~75–80% past here, a 5% traffic rise doubles response time 50% → 2× 80% → 5× 90% → 10× 95% → 20×
Utilisation targets are latency decisions in disguise. Running a single-threaded resource — one CPU, one connection, one partition consumer — at 90% is choosing a tenfold wait. Variance makes it worse: bursty arrivals reach the knee locally even when the average looks calm, which is why the average CPU graph can read 45% while p99 is already suffering.
Part two

Concurrency, in depth

Two things go wrong with concurrent code: threads disagree about what memory says, and threads queue behind each other. The first is a correctness problem with a precise rule; the second is a scaling problem with a precise curve.

Foundations deep · 04

Memory visibility and the happens-before edge

A write by one thread is not automatically a write the others can see. Compilers reorder, CPUs buffer, caches lag. Correctness comes from establishing an edge — a synchronising operation that both sides participate in.

NO EDGE — UNDEFINED, NOT MERELY UNLIKELY thread A thread B data = 42 ready = true if ready { reads 0 B can see ready without data: the two stores may reach B in either order, or the loop may be hoisted out entirely and never re-read at all. WITH AN EDGE — EVERYTHING PRIOR IS PUBLISHED data = 42 unlock / send lock / receive reads 42 happens-before The edge is a one-way barrier: writes before the release are visible after the acquire. It costs tens of nanoseconds and it is the entire reason locks exist. Where edges come from mutex unlock → lock · channel send → receive · atomic store-release → load-acquire · goroutine/thread start · WaitGroup Wait · Once.Do Nothing else. A bool flag, a sleep, or “it worked when I tested it” are not edges.
This is why a race detector finds bugs your reasoning cannot. The broken version on the left works almost always on x86 and fails on ARM, under a new compiler, or once the loop gets hot enough to optimise. Correctness here is not probabilistic — without an edge, the program simply has no defined meaning.
Foundations deep · 05

Contention, coherency, and the curve that turns down

Adding workers buys less than you expect (Amdahl) and eventually buys negative throughput (the Universal Scalability Law). The second effect surprises people, and it is the reason a bigger instance sometimes performs worse.

CONCURRENT WORKERS → THROUGHPUT linear ideal — never observed Amdahl — a serial section caps you peak USL — coordination costs turn it down
Three regimes, one system. Contention (waiting for a shared resource) flattens the curve; coherency (workers having to agree — cache-line ping-pong, lock handoffs, distributed consensus) bends it down. Past the peak, every extra worker makes the system slower. Finding that peak empirically is more useful than any theoretical target.

Symptoms you can measure

Throughput flat while CPU climbs → contention. Throughput falling as you add pods or threads → coherency: you are paying more to coordinate than you gain. Lock wait time and context-switch rate are the two counters that separate them.

The fix is structural, not tuning

Shrink the critical section, shard the lock by key, or remove the sharing entirely with a single-writer-per-partition design. Faster locks buy you a constant factor; removing the shared point changes the shape of the curve.

Part three

Events, in depth

Asynchronous systems trade one hard problem (availability coupling) for two others: what "delivered" means, and what happens when step three of five fails after steps one and two already committed.

Foundations deep · 06

Delivery semantics are just where you put the ack

There is no magic in "exactly once". The three semantics differ only in the order of two operations — acknowledging the message and doing the work — plus, optionally, a record of what you have already done.

A · ACK FIRST — AT MOST ONCE receive ACK do the work crash → message gone forever Only correct for data you can afford to lose: metrics, presence pings, sampling. B · ACK LAST — AT LEAST ONCE (the default everywhere) receive do the work ACK crash → redelivered → work runs twice Nothing is lost. The card is charged twice. Not acceptable on its own. C · ACK LAST + LEDGER — EFFECTIVELY ONCE receive INSERT event_id + do work, one TX ACK redelivery hits a duplicate key → no-op Delivery is still at-least-once. The effect happens once, because the ledger and the work share a transaction.
Row C is the whole trick, and it needs the ledger and the side effect in one atomic unit. If the dedup table lives in Redis and the work lands in Postgres, you have re-created the dual-write problem one level down. Where the effect is external (a payment API), push the idempotency key to them — a provider-side key is the only thing that makes a remote call safe to retry.
Foundations deep · 07

Sagas: what happens when step three fails

A distributed transaction cannot roll back — steps one and two are already committed in other systems. A saga replaces rollback with compensation: a new, forward action that undoes the business effect of an earlier one.

FORWARD PATH 1 reserve stock committed ✓ 2 charge card committed ✓ 3 book courier FAILS — no capacity 4 notify customer never runs COMPENSATING PATH — REVERSE ORDER, FORWARD ACTIONS C2 refund card a new charge, not an undo C1 release stock back to the pool compensations fail too retry them forever, then page a human — never drop them What the customer sees is part of the design Between step 2 and C2 the money really has left their account. The saga is correct and the experience is still bad, so the product answer — authorise instead of capture, or hold the order in “pending” — is chosen at the same time as the technical one. Every step needs: an idempotency key, a compensation, and a timeout after which the saga decides on its own.
Sagas are easy to draw and hard to operate. The state of every in-flight saga must be durable (a table, a workflow engine, a state machine in the log), because the coordinator can crash between any two boxes here. That persistence — not the happy path — is what you are really choosing when you pick orchestration over choreography.
Part four

Caching, in depth

Two things are worth knowing beyond "store the answer": the arithmetic that tells you what a hit ratio is actually worth, and the exact interleavings that leave a wrong value sitting in the cache.

Foundations deep · 08

The arithmetic of a hit ratio

Effective latency is h · Lhit + (1−h) · Lmiss, and origin load is (1−h) · λ. Both are dominated by the miss term, which is why the last percent of hit ratio is worth more than the first ninety.

10,000 rps · cache hit 1ms · origin 40ms h = 90% 4.9ms average · 1,000 rps hits the origin h = 95% 2.95ms average · 500 rps hits the origin h = 99% 1.39ms average · 100 rps hits the origin hit time ← time spent on the misses Two consequences people get wrong 90% → 99% is a 10× reduction in origin load, not a 10% improvement. The interesting number is the miss rate, so track (1−h) on a log axis and you will see incidents that a hit-ratio dashboard hides. The average is not the experience. At h = 99%, one request in a hundred takes 40ms — so the cache does nothing for p99 by construction. Caches fix load and the median; only a faster origin fixes the tail.
The bar lengths are the miss cost, drawn to scale. Notice what the origin must survive: at h = 90% it is sized for 1,000 rps, so a cache outage multiplies its load by ten instantly. The cache's job description therefore includes "what the origin does when I disappear".
Foundations deep · 09

The invalidation race, drawn

Cache-aside has one interleaving that quietly stores a value that was already stale when it was read. It is rare per request and certain at scale, and no amount of careful ordering in the writer removes it.

TIME → reader (slow) writer (fast) 1 MISS 2 read DB → v1 3 commit v2 4 DEL key 5 SET key = v1 the reader was still in flight this whole time Cache now serves v1 until the TTL expires. Four mitigations, in order of cost 1. A short TTL bounds the damage — the cheapest fix, and usually enough. 2. Delete after commit, then again after a delay catches readers that were mid-flight. 3. Version the value and refuse to overwrite a newer one (CAS or a per-key sequence). 4. Write through a single owner — only one code path may populate the key, so there is no race left to lose.
Note what this race is not: it is not fixed by deleting before the commit instead of after (that leaves a window where the old value is re-cached from an uncommitted read), and it is not fixed by writing the new value into the cache instead of deleting (same interleaving, same outcome). Accept a bounded staleness window or make writes ordered per key.
Part five

Storage, in depth

Every storage engine spends the same three currencies: read work, write work, and space. You cannot minimise all three, and knowing which one a design is spending explains most of its behaviour under load.

Foundations deep · 10

Read, update, memory — pick two

Amplification is the ratio between the work you asked for and the work the disk actually did. A B-tree pays it on writes; an LSM pays it on reads and on background compaction; an index pays it in space and on every write to the table.

THE THREE CURRENCIES READ UPDATE MEMORY B-tree: cheap reads, expensive random writes LSM: cheap writes, read + space cost 6 indexes: fast queries, slow writes Moving toward one corner moves away from another. ONE ROW UPDATE, TWO WRITE PATHS B-TREE — UPDATE IN PLACE WAL append find leaf page rewrite 8KB page A 200-byte change costs an 8KB page write, plus one per secondary index, at a random offset. Reads stay one descent. LSM — APPEND, SORT LATER WAL append memtable flush L0 compact L1..Ln The write returns after the memtable. Everything after that is background work — which is why LSM latency is smooth until compaction falls behind, and then is not. A read may touch several levels; bloom filters skip the ones that cannot contain the key, which is what makes LSM point reads competitive at all.
Pick the engine from the write pattern, not the read one. Ingest-heavy, append-shaped workloads (events, metrics, feeds) suit LSM; update-heavy transactional rows with range scans suit a B-tree. The failure modes differ too: a B-tree degrades gradually through fragmentation, an LSM degrades suddenly when compaction cannot keep up and reads start touching every level.

The index count is a write-throughput decision

Each secondary index is another structure to update on every insert and on any update touching its columns. Six indexes on a hot table means one logical write becomes seven physical ones, at random offsets. When write latency creeps up with no code change, count the indexes that were added "just for that one report".

Part six

Load and failure, in depth

Three mechanisms decide whether an overloaded system degrades or collapses: whether pressure propagates backwards, whether deadlines shrink as they travel, and whether the failure sustains itself after the trigger is gone.

Foundations deep · 11

Backpressure, drawn end to end

Backpressure is the property that a slow component makes its caller slow down, all the way to the entry point. Its absence is not neutral — unbounded buffers convert a throughput shortfall into unbounded latency and then into an out-of-memory kill.

NO BACKPRESSURE — EVERY BUFFER UNBOUNDED clients LB queue accept q worker ch slow DB Nobody is told to stop. Queues absorb the shortfall as memory and as latency, until the process is OOM-killed — and every queued request has already timed out at the client anyway. queues grow — work that nobody is waiting for any more WITH BACKPRESSURE — EVERY BUFFER BOUNDED clients LB · max 1k accept 512 chan cap 64 pool of 20 “full” travels backwards in milliseconds → 429 at the edge The system sheds 8% of traffic and serves the rest at normal latency, instead of serving 100% of it badly — or none of it. Rule of thumb: a queue you cannot name the maximum size of is a memory leak with a scheduler attached.
Bounds are only half of it — the other half is dropping work nobody wants. A queued request whose client already timed out is pure waste that makes the recovery slower. Check the deadline before starting work, and prefer LIFO over FIFO while overloaded: it serves the newest requests, which are the only ones still being waited on.
Foundations deep · 12

Timeout budgets and availability arithmetic

Two numbers travel down a call tree and must be managed explicitly: the time remaining, which shrinks at every hop, and the probability of success, which multiplies at every hop.

THE DEADLINE SHRINKS AS IT TRAVELS client budget 300ms gateway 280ms left orders svc 250ms left cache · 20ms cap db · 120ms cap pricing · 80ms cap 20 + 120 + 80 = 220ms of caps inside a 250ms budget, leaving 30ms for one retry and the service’s own work. If the caps sum above the budget, the client gives up first — and everything downstream keeps working on a dead request. AND THE SUCCESS PROBABILITY MULTIPLIES 99.9% 99.9% 99.9% 99.9% = 99.6% — 3.5 hours of downtime a year SERIAL — all four must work PARALLEL — either one will do 99.9% 99.9% = 99.9999% — if the failures are truly independent Independence is the assumption that fails: same AZ, same config push, same bad deploy.
Both halves argue for the same design move: fewer things must succeed. Make a dependency optional and it leaves the serial chain entirely — the pricing call above, given a cached fallback, stops being able to fail the request at all. That is worth more than any amount of retry tuning.
Foundations deep · 13

Metastable failure: when removing the cause doesn’t help

The hardest outages have this shape: a trigger pushes the system into a state that sustains itself, so the system stays down after the trigger is gone. Adding capacity does not fix it, and restarting often makes it worse.

STABLE 60% load, warm cache TRIGGER deploy, spike, failover METASTABLE down, and staying down trigger passes… but nothing recovers THE SUSTAINING LOOP timeouts → retries 3× the offered load everything slower still caches cold, pools full The only exit is less work, not more capacity Shed load hard, disable retries, drop the queue, let caches refill, then admit traffic back gradually. Scaling up during a metastable failure often deepens it: new instances start cold, miss every cache, and add load to the thing that is already saturated.
The tell is that recovery does not follow the trigger. Traffic is back to normal, the bad deploy is rolled back, CPU is not maxed — and the error rate stays pinned. Prevention is retry budgets, circuit breakers and admission control; the recovery lever is a switch you built in advance, because you will not write one during the incident.
Part seven

Put it together

One worked example that uses every tool on this page. This is the shape of answer a systems-design interview is looking for, and the shape of a capacity document that survives review.

Foundations deep · 14

Worked example: size this service

The brief: an order-read API, 3,000 requests per second at peak, p50 of 40 ms and p99 of 120 ms, backed by Postgres and a Redis cache. How many pods, how many connections, what timeouts, and what breaks first?

1 · IN FLIGHT L = 3,000 × 0.040 = 120 concurrent 2 · HEADROOM target ρ = 0.6, not 0.9 plan for 200 3 · PODS ~25 in flight per pod 8 pods, 3 AZs + N-1 RULE one AZ down = 2/3 capacity, so 8 must cope at 6 4 · CONNECTIONS — THE REAL CONSTRAINT 15ms of the 40ms is spent in Postgres L(db) = 3,000 × 0.015 = 45 busy connections 8 pods × 10 = 80 — fine 8 pods × 50 = 400 — exceeds max_connections 5 · WHAT BREAKS FIRST Not CPU. The pool: if DB latency triples to 45ms, L(db) = 135 > 80, so requests queue for a connection and p99 leaves the budget. 6 · THE BRAKES Pool wait timeout 50ms → fail fast Query timeout 100ms → bound W HPA on in-flight, not CPU Breaker + cached fallback The answer that scores: “8 pods, 10 connections each, and the pool is the bottleneck.” Naming the constraint and the failure mode is the senior part. The pod count is arithmetic; knowing that a slow dependency — not more traffic — is what will take this service down is the judgement the role is hiring for.
Every step here is one of the earlier sections applied. Little's Law gives concurrency, the utilisation curve gives the headroom multiplier, connection maths finds the true bottleneck, and the brakes are the fault-tolerance patterns chosen for the specific way this design fails.
  1. Start from throughput and latency, never from instance sizes. Concurrency is derived, not chosen.
  2. Pick a utilisation target before counting boxes — 60–70% for latency-sensitive work, higher only for batch where queueing is free.
  3. Find the scarcest serialised resource. Connections, partitions, a single-writer shard, a third-party rate limit. That is your real capacity, whatever CPU says.
  4. Survive N-1. Size so the remaining zones absorb the load; otherwise an AZ failure is an outage with extra steps.
  5. Give every queue a bound and every wait a timeout, then decide what the system does when it hits them — degrade, shed, or fail.
  6. Write down what breaks first and at what number. An estimate you can falsify is worth more than a precise one you cannot.

The four questions, restated for this page

What is the queue? Every stage has one, even when it is only a connection pool. How full is it? Utilisation, not capacity, sets latency. What happens when the thing behind it slows down? In-flight work grows by Little's Law until something bounds it. Does the failure sustain itself? If retries and cold caches can feed it, you need a shedding lever, not a bigger cluster.