The Backend Ladder
0%
Three requirements · three depths each

Deep backend
foundations, and
enough Go to be
dangerous.

A job description asks for three things: systems thinking that survives contact with production, Go fluency good enough to read, review and write real code, and hands-on comfort with the cloud those systems actually run on. This page walks all three from beginner to advanced — mostly in pictures — and then puts every concept on the same map.

Pillar A

Deep Backend Foundations

Concurrency · event-driven architecture · caching · indexing · message queues · fault tolerance. Language-agnostic on purpose.

Pillar B

Pragmatic Go

Goroutines · channels · interfaces · error handling. Enough to read a diff, catch a bug in review, and write the fix.

Pillar C

Cloud & Infrastructure

AWS accounts, networks and managed services · Kubernetes objects, health and scaling. The same foundations, bought and configured.

Orientation

What each level actually means

The three levels here are not "how much you have read". They are three different relationships with the same idea: you can name it, you can operate it, or you can own the tradeoff it makes. Every topic below is split the same way, so the ladder is comparable across concepts.

L1 · BEGINNER Name the mechanism “A cache stores a copy to avoid work.” L2 · INTERMEDIATE Operate it under load “Cache-aside, TTL 60s, keyed by version.” L3 · ADVANCED Own the tradeoff “What happens when it’s cold at 09:00?” + knows the knobs + knows the failure DEPTH → answers “what breaks, and what did we buy by accepting that?”
The ladder used throughout this page. The jump from L1 to L2 is knobs and operations. The jump from L2 to L3 is failure modes and the price paid — which is the level most interviews and most on-call pages actually probe.
Pillar A

Deep backend foundations

Six mechanisms that show up in every backend, regardless of language. Each one exists to trade a resource you have for a resource you do not: CPU for latency, memory for reads, durability for throughput, correctness for availability.

Deep dive: The Machine Room →

Foundations · 01

Concurrency

Concurrency is a way of structuring work so parts can progress independently. Parallelism is executing them at the same instant. You can have concurrency on one core; you get parallelism only with more cores.

A · CONCURRENT, 1 CORE B · PARALLEL, 2 CORES C · THE RACE task 1 task 2 · time → Both in flight. Only one running. Wins on I/O-bound work. core 0 core 1 · time → Same clock, real speedup. Wins on CPU-bound work. T1: read n = 7 T2: read n = 7 T1: write n = 8 T2: write n = 8 Two increments, one lost. `n++` is read–modify–write, not one step. The whole subject in one line: any state reachable by two threads at once needs a rule — a lock, an atomic, a single owner, or a copy.
A and B are different questions: concurrency is about structure, parallelism about hardware. C is why the structure needs rules — the interleaving is legal, the result is wrong, and it reproduces once a week at 3am.
L1 · Beginner

Know the vocabulary and the hazard

  • Process vs thread — separate memory vs shared memory, and why sharing is what creates the problem.
  • Race condition — result depends on timing. Critical section — the code that must not interleave.
  • Mutex to serialise access; atomics for single-word counters.
  • Context switch costs; blocking on I/O is what makes concurrency pay off.
  • CPU-bound vs I/O-bound — the first wants cores, the second wants overlap.
L2 · Intermediate

Coordinate safely at scale

  • Deadlock and the four conditions; break it with a global lock ordering.
  • Lock granularity — coarse locks are correct and slow; fine locks are fast and easy to get wrong. Sharded locks as the middle.
  • Memory visibility — without a happens-before edge, another thread may never see your write.
  • Bounded concurrency: worker pools, semaphores, and why unbounded goroutines/threads is a production incident.
  • Backpressure — a full queue must slow the producer, not silently grow.
  • Cancellation propagated through the whole call tree.
L3 · Advanced

Reason about the scaling ceiling

  • Amdahl & the Universal Scalability Law — contention flattens the curve, coherency (cross-core chatter) bends it back down. More cores can be slower.
  • False sharing — two hot variables on one cache line silently serialise.
  • Lock-free via CAS, and knowing when it is not worth it: uncontended mutexes are already cheap.
  • Single-writer principle — partition state by key so no lock is needed at all. Usually beats clever locking.
  • Structured concurrency — no goroutine outlives the scope that spawned it; leaks become impossible by construction.
  • Latency under contention is a distribution: watch p99, not the mean.
1 · SHARE + LOCK w1 w2 w3 lock state Correct, familiar. Throughput ceiling = time inside the lock. 2 · SEND TO ONE OWNER w1 w2 w3 chan queue owner+state No lock; the queue is the serialisation — and it must be bounded. 3 · PARTITION BY KEY w1 w2 w3 shard A shard B shard C Nothing shared, nothing to coordinate. Fails only on a hot key. Same problem, three prices: option 1 pays in contention, option 2 pays in queueing delay, option 3 pays in key skew.
The comparison that matters. These are the only three answers to shared mutable state. Senior judgement is picking by which price you can afford — and noticing that partitioning (3) removes the problem rather than managing it, which is why it scales furthest.
Foundations · 02

Event-driven architecture

A command asks for something to happen and waits. An event announces something already happened and does not care who listens. Swapping one for the other changes who is coupled to whom, and who is awake when things break.

SYNCHRONOUS ORCHESTRATION EVENT-DRIVEN CHOREOGRAPHY Orders API Payments Inventory Email Analytics blocking calls Latency adds up. Availability multiplies down. 4 × 99.9% = 99.6%. Email being down fails the order. Adding a 5th consumer means changing Orders. Upside: you know the answer before replying. Orders API append-only log 1 write, then reply OrderPlaced Payments Inventory Email Analytics each reads at its own pace Orders survives all four being down. New consumer = no change to Orders. Cost: no answer yet. Debugging spans hops.
The difference is the edge count and its direction. On the left, Orders depends on four services. On the right it depends on one log, and the four depend on the log's schema. You have not deleted coupling — you have moved it from runtime availability to a data contract.
L1 · Beginner

Events, publishers, subscribers

  • An event is a fact in the past tense: OrderPlaced, not PlaceOrder. It cannot be rejected.
  • Pub/sub vs point-to-point; the publisher does not know its subscribers.
  • Sync gives you an answer; async gives you decoupling. That is the whole trade.
  • Asynchronous means the caller returns before the work is done — so the UI must be designed for "in progress".
L2 · Intermediate

Contracts, ordering, and the dual-write trap

  • Choreography vs orchestration — events reacting to events, vs one coordinator driving steps. Orchestration is easier to debug; choreography is easier to extend.
  • Three payload styles: notification (id only), event-carried state transfer (full snapshot), event sourcing (the log is the truth).
  • Idempotent consumers — assume every event arrives twice. Dedupe on an event id.
  • Ordering is per partition key, never global. Key by aggregate id.
  • Dual write: writing the DB and publishing separately can lose or invent events → fix with the transactional outbox or CDC.
L3 · Advanced

Truth, time, and recovery

  • Event sourcing + CQRS — rebuild any read model by replaying; the cost is versioning every event forever.
  • Schema evolution: additive-only fields, registries, and a compatibility policy you enforce in CI.
  • Sagas — distributed transactions as steps plus compensations; there is no rollback, only a corrective event.
  • Effectively-once = at-least-once delivery + idempotent effects. "Exactly once" end-to-end is marketing.
  • Eventual consistency in the product: read-your-writes, stale reads, and what the user is told meanwhile.
  • Replay safety — can you re-run last week's log without re-charging a card?
BROKEN · DUAL WRITE service DB ✓ broker 1. commit 2. publish — crash here Order exists. Nobody was told. Two systems, one intent, no shared transaction. Retrying first can equally publish twice with no commit. FIXED · TRANSACTIONAL OUTBOX service ONE TRANSACTION orders row outbox row relay / CDC tail polls broker at-least-once A crash now leaves an unsent row, which the relay retries. The consumer is idempotent.
The single most useful pattern in this topic. The fix is not a better broker — it is refusing to write to two systems at once. State and intent go into the same transaction; a separate process turns intent into a message, and is allowed to be slow or repeat itself.
Foundations · 03

Caching

A cache buys latency and load relief with memory and staleness. Every cache is a bet that the same answer will be asked for again before it stops being true — and a new failure mode for the moment the bet is wrong.

A · CACHE-ASIDE: THE READ PATH client app cache DB 1. GET 2a. HIT → return (~1ms) 2b. MISS 3. read DB, SET with TTL (~40ms) B · WRITES: INVALIDATE, DON’T UPDATE writer DB cache 1. commit 2. DEL key Delete is idempotent and order-independent; writing the new value races with a concurrent slow reader that is about to SET an older one. C · THE STAMPEDE t = TTL expiry 10k in-flight requests all miss the same key in the same millisecond. The DB gets the full unfiltered load it was never sized for. THREE FIXES coalesce — one loader per key, others wait jitter the TTL so expiries spread out refresh early, probabilistically, before expiry
Cache-aside is the default for a reason: the app owns the policy, and a cache outage degrades to slow rather than broken. The expensive lessons are on the right — a cache that removes 95% of your database load has also made your database unable to serve 100% of it, which is exactly what it must do the moment the cache blinks.
L1 · Beginner

What, where, how long

  • Why: cut latency and cut load on something expensive.
  • Layers: browser → CDN → in-process → distributed (Redis/Memcached) → DB buffer pool.
  • Hit ratio, TTL, eviction (LRU, LFU) and what happens when memory fills.
  • Cache only what is expensive and re-read. Everything else is a memory leak with extra steps.
L2 · Intermediate

Patterns and key design

  • Cache-aside, read-through, write-through, write-behind — and which one loses data on a crash.
  • Key design: namespace, entity, version. Bumping a version prefix invalidates a whole class instantly.
  • Negative caching for "not found", or one bad id becomes a DB scan generator.
  • Local (fast, per-instance, inconsistent) vs distributed (shared, consistent-ish, a network hop away).
  • Measure hit ratio and the latency of a miss — a 99% hit ratio with a 2s miss is still a bad p99.
L3 · Advanced

The cache as a liability

  • Stampede / dogpile → single-flight coalescing, TTL jitter, probabilistic early recompute.
  • Cold start — a deploy or Redis failover exposes the origin to unfiltered traffic. Warm, or shed load.
  • Hot keys — one celebrity id saturating one shard; fix with key splitting or a local L1 in front.
  • Consistent hashing so adding a node moves 1/N of keys, not all of them.
  • Write–invalidate races; fix with delete-after-commit, short TTLs as a safety net, or versioned CAS writes.
  • Decide explicitly: is stale data wrong here, or merely old? Prices and permissions differ from avatars.
Foundations · 04

Indexing

An index is a second, ordered copy of some columns that turns "look at everything" into "navigate to it". You pay on every write and in storage, forever, to make certain reads cheap.

WITHOUT INDEX · SEQUENTIAL SCAN reads every page · O(n) · 10M rows → ~4s WITH B-TREE INDEX · DESCENT root · <500k 0–250k 250k–500k leaf → row ptr 3–4 page reads · O(log n) · 10M rows → ~1ms Depth grows with the logarithm: 10× the rows is roughly one extra hop. COMPOSITE INDEX · LEFTMOST PREFIX INDEX (tenant_id, status, created_at) sorted by column 1, then 2, then 3 — like a phone book ✓ WHERE tenant=7 ✓ WHERE tenant=7 AND status='new' ✓ ...AND created_at > '2026-01-01' ✓ ORDER BY created_at (free sort) ✗ WHERE status='new' skips column 1 → index unusable ✗ WHERE tenant=7 AND created_at > X stops at the gap; filters the rest by hand Column order is the design decision, not an implementation detail.
Left: why an index exists. Right: why the wrong one does nothing. A composite index is one sorted list, so it can only be entered from the left — and only the first ranged column can be followed by useful equality matches. Reading a query plan is the skill that makes this concrete.
L1 · Beginner

Lookup instead of scan

  • Index = ordered structure mapping a value → row location. B+tree is the default.
  • Primary key vs secondary index; unique vs non-unique.
  • The cost: every INSERT/UPDATE maintains every index. Indexes are not free reads — they are bought with writes.
  • Index the columns you filter and join on, not every column.
L2 · Intermediate

Read the plan, design the key

  • Leftmost prefix and column ordering: equality columns first, the range column last.
  • Selectivity — an index on a boolean rarely helps; the planner will ignore it and be right.
  • Covering index / index-only scan — put the returned columns in the index and skip the table entirely.
  • Read EXPLAIN ANALYZE: estimated vs actual rows is where bad plans confess.
  • Structure per job: B-tree (range), hash (equality), inverted (search), GiST/geo, LSM (write-heavy).
L3 · Advanced

Write amplification and plan stability

  • B-tree vs LSM — in-place updates and read-optimised, vs sequential writes with compaction and read amplification. Bloom filters recover point reads.
  • Clustered (row lives in the index, InnoDB) vs heap (Postgres) — changes the cost of every secondary lookup.
  • Partial and expression indexes — index the 2% of rows a hot query touches.
  • Bloat, vacuum, and write cliffs — index maintenance is an operational concern, not a one-time DDL.
  • Cardinality misestimates flip plans overnight under stale statistics; that is the 3am "the query got slow with no deploy".
  • At scale the partition/shard key outranks every index: it decides which machine even has to answer.
Foundations · 05

Messaging queues

A queue converts a synchronous dependency into a buffer with a time dimension. Producers stop caring whether consumers are up, fast, or numerous — and in exchange you now own a backlog, a retry policy, and a poison message somewhere.

PRODUCERS PARTITIONED LOG · ORDER HOLDS WITHIN A PARTITION CONSUMER GROUP api-1 api-2 api-3 partition = hash(key) % N P0 P1 P2 offset LAG on P1 = 3 — the number to alert on c-1 c-2 c-3 max useful consumers = N DLQ after n retries Four numbers describe any queue in production: lag (how far behind), age of oldest message (how stale), redelivery rate (how broken), DLQ depth (how much needs a human). Throughput alone tells you nothing — a consumer that fails instantly has excellent throughput.
Read this diagram left to right and the whole topic follows. The partition key decides ordering and parallelism at once: same key → same partition → strict order but one consumer. More partitions buys concurrency and costs global ordering. Everything else — lag, retries, the DLQ — is bookkeeping around that one choice.
L1 · Beginner

Producer, broker, consumer

  • Why: decouple, buffer spikes, retry safely, do slow work off the request path.
  • Queue (one consumer wins) vs topic (everyone gets a copy).
  • Ack — the broker keeps a message until the consumer confirms; no ack means redelivery.
  • Producing is fast; the work still has to happen. A queue moves latency, it does not delete it.
L2 · Intermediate

Semantics you can defend

  • At-most-once / at-least-once / effectively-once. Nearly everything real is at-least-once, so consumers must be idempotent.
  • Consumer groups and partitions: parallelism is capped by partition count.
  • Visibility timeout / ack deadline — too short and you double-process, too long and failures stall.
  • DLQ plus a plan for draining it. A DLQ nobody reads is a bug tracker nobody reads.
  • Lag is the health metric; alert on it before latency SLOs break.
L3 · Advanced

Topology and failure economics

  • Log-based (Kafka/Pulsar: retained, replayable, ordered per partition) vs broker-based (SQS/Rabbit: per-message ack, easy fan-out of work, no replay). Pick by whether you need history.
  • Hot partitions from a skewed key; rebalancing storms when consumers flap.
  • Poison messages blocking an ordered partition — ordering guarantees and skip-on-failure are in direct conflict.
  • Backpressure and flow control: an unbounded queue converts a throughput problem into an unbounded latency problem.
  • Replay as an operational tool — and the idempotency that makes it safe.
  • Queue depth as a control signal for autoscaling, not just a dashboard line.
Foundations · 06

Fault tolerance

Everything downstream will be slow before it is down, and down before you notice. Fault tolerance is the set of decisions you make in advance about what to do with the time you are not going to get an answer in.

ONE OUTBOUND CALL, FIVE LAYERS OF DEFENCE caller 1 timeout 300ms 2 retry ×2 +jitter 3 breaker closed 4 bulkhead max 20 dependency 5 fallback: cached / partial on open the caller’s own deadline bounds all of it CIRCUIT BREAKER · STATE MACHINE CLOSED OPEN HALF-OPEN error rate > 50% after cool-off probe succeeds → close probe fails → re-open OPEN fails instantly — which frees threads and gives the dependency room to recover. WHY RETRIES NEED A BUDGET normal load ×3 from one retrying tier ×9 when two tiers both retry Retries multiply through layers. Retry at one layer, with a cap.
These layers are ordered, not interchangeable. A timeout without a retry drops work; a retry without backoff and a cap is a load amplifier; a breaker without a fallback just fails faster. And every one of them is only safe if the operation is idempotent — which is why idempotency keys belong in the design, not the postmortem.
L1 · Beginner

Assume failure

  • Networks drop, disks fill, processes are killed mid-write. Failure is normal traffic.
  • Always set a timeout. An unset client timeout is an unbounded outage waiting for a slow day.
  • Redundancy: more than one instance, health checks, restart on failure.
  • Graceful degradation — return the page without the recommendations rather than no page.
L2 · Intermediate

Patterns with knobs

  • Exponential backoff + jitter, capped attempts. Jitter is what stops synchronised retries.
  • Idempotency keys so a retry cannot charge twice.
  • Circuit breaker (closed/open/half-open) and bulkheads so one slow dependency cannot consume every worker.
  • Rate limiting and load shedding — rejecting 5% cleanly beats collapsing for 100%.
  • Deadline propagation — the whole call tree shares one budget, decremented at each hop.
L3 · Advanced

Systemic failure

  • Metastable failure — the trigger passes but retries and cold caches keep the system down. Recovery needs load shedding, not more capacity.
  • Blast radius: failure domains, cells, shuffle sharding, so one bad tenant cannot reach everyone.
  • Gray failure — healthy to the health check, broken for users. Probe what users do.
  • Hedged requests to cut tail latency, paid for in extra load.
  • CAP / PACELC as a design statement: during a partition, do you refuse writes or accept divergence? Say which, on purpose.
  • SLOs and error budgets turn "reliable" into a number that decides whether you ship or harden.
Pillar B

Pragmatic Go

Go is a small language with four ideas that carry almost everything: goroutines, channels, interfaces, and errors as values. The bar here is fluency for reading and reviewing — being able to look at a diff and say "that goroutine leaks", "that interface is nil but not nil", "that error swallows context".

Deep dive: Reading Go →

Go · 01

Goroutines

A goroutine is a function scheduled by the Go runtime rather than the OS: a few kilobytes of stack, started with one keyword. Cheap enough that "one per request" is idiomatic — which is exactly why leaking them is the most common Go production bug.

100,000 GOROUTINES (G) GOMAXPROCS LOGICAL CPUS (P) OS THREADS (M) ~2KB stack each, grown on demand created by `go f()` in ~nanoseconds run queues P0 P1 · idle P2 idle P steals half of a busy P’s queue thread M0 thread M1 thread M2 netpoller (epoll/kqueue) socket waits park the G, not the thread a blocking syscall detaches M; a new M takes over the P so work continues Why it matters in review: goroutine-per-request is fine, but every goroutine needs an owner who knows when it ends — otherwise the count only goes up.
The G–M–P model, and why blocking I/O in Go feels free. Goroutines are multiplexed onto a fixed number of logical processors; the netpoller means an idle socket costs a parked goroutine rather than a parked thread. Work stealing keeps every P busy without you scheduling anything by hand.
// The leak: nobody ever receives, so this goroutine blocks forever.
// Its memory, its stack, and everything it closes over stay alive.
func leak() {
    ch := make(chan int)          // unbuffered
    go func() { ch <- expensive() }()  // blocks on send, forever
    return                        // caller walked away
}

// The fix: an exit path the goroutine can actually take.
func noLeak(ctx context.Context) (int, error) {
    ch := make(chan int, 1)       // buffered: send never blocks
    go func() { ch <- expensive() }()
    select {
    case v := <-ch:
        return v, nil
    case <-ctx.Done():            // caller gave up; we still unblock
        return 0, ctx.Err()
    }
}
L1 · Beginner

Start one, wait for it

  • go f() starts a goroutine; the caller does not wait.
  • sync.WaitGroup to wait for a known set. Add before go, defer wg.Done() inside.
  • When main returns, every goroutine dies mid-sentence. Shut down on purpose.
  • Run tests with -race. It finds real bugs you cannot reason your way to.
L2 · Intermediate

Lifetimes and limits

  • context.Context is the cancellation and deadline channel of every real Go service. First parameter, always.
  • Bounded worker pools — N workers over one job channel, or a buffered channel as a semaphore. Unbounded fan-out is how you OOM.
  • errgroup.Group — wait for a group, propagate the first error, cancel the rest.
  • Leak review checklist: every go has a stop condition; every send has a receiver or a ctx.Done().
  • Loop-variable capture is per-iteration since Go 1.22 — know which version the codebase is on.
L3 · Advanced

Runtime behaviour

  • Work stealing, preemption, and the netpoller — why blocking I/O scales but a tight CPU loop used to starve everything.
  • GOMAXPROCS in containers — unset, Go can see host cores and oversubscribe a small CPU quota (fixed by newer runtimes / automaxprocs).
  • Diagnosis: pprof goroutine profiles, goleak in tests, the runtime trace for scheduler latency.
  • Structured concurrency — the caller owns the scope; nothing outlives it.
  • sync.Pool and allocation pressure; GC and goroutine count are related costs.
Go · 02

Channels

A channel is a typed, synchronised handoff. Unbuffered channels are a rendezvous: the send does not complete until someone receives. Buffered channels are a bounded queue — and that bound is the backpressure from the foundations section, expressed in one integer.

UNBUFFERED · make(chan T) sender receiver sender blocked, waiting value moves when both are ready A synchronisation point: after the handoff, everything the sender did is visible to the receiver. Zero capacity means no hidden queue. BUFFERED · make(chan T, 3) sender receiver 2 of 3 used — sends return immediately Full buffer → the sender blocks. That block is your backpressure. Sizing the buffer is choosing how much burst you absorb before pushing back. Capacity N also makes a clean semaphore for N-way limits. SELECT · FIRST READY WINS select result ctx.Done timeout Ready more than one? Picked at random, so no case can starve. The three rules that prevent most channel bugs 1. The sender closes, never the receiver — and only when no other sender remains. 2. A receive from a closed channel returns the zero value immediately; use v, ok := <-ch to tell them apart. 3. A nil channel blocks forever — which is the idiomatic way to disable one case of a select in a loop.
Buffered or not is a semantic choice, not a performance tweak. Unbuffered gives you a guarantee that the other side received; buffered gives you burst absorption and a defined point where you push back. Choosing a buffer size is choosing your queueing policy.
// Bounded fan-out: 8 workers, backpressure built in, cancellable.
func process(ctx context.Context, jobs <-chan Job) error {
    g, ctx := errgroup.WithContext(ctx)
    for i := 0; i < 8; i++ {          // fixed concurrency, not one per job
        g.Go(func() error {
            for {
                select {
                case j, ok := <-jobs:
                    if !ok {           // channel closed: clean exit
                        return nil
                    }
                    if err := handle(ctx, j); err != nil {
                        return err     // cancels ctx for the whole group
                    }
                case <-ctx.Done():     // someone else failed, or caller gave up
                    return ctx.Err()
                }
            }
        })
    }
    return g.Wait()                    // first error, after all workers stop
}
L1 · Beginner

Send, receive, close

  • ch <- v sends, v := <-ch receives; both block until the other side is ready.
  • make(chan T) vs make(chan T, n).
  • close(ch) then for v := range ch to drain; sending on a closed channel panics.
  • Directional types in signatures — <-chan T and chan<- T document intent and are checked by the compiler.
L2 · Intermediate

Compose them

  • select with ctx.Done() is the shape of nearly every production loop.
  • default: makes a non-blocking try — useful for drop-on-full metrics, dangerous as a busy loop.
  • Pipelines, fan-out/fan-in, and closing a downstream stage only after every upstream sender has finished.
  • Buffered channel of capacity N as a semaphore around an expensive resource.
  • time.After in a hot loop allocates a timer per iteration; prefer time.NewTimer and stop it.
L3 · Advanced

Know when not to

  • "Share memory by communicating" is a default, not a law. For a hot counter or a shared map, sync.Mutex or sync/atomic is simpler and faster — a channel op costs far more than an uncontended lock.
  • Unbuffered send/receive establishes a happens-before edge; that is the memory-model reason channels are safe.
  • Deadlock shapes: circular pipelines, sending to yourself, and waiting on a channel nobody will close.
  • Every channel needs a documented owner: who sends, who closes, who drains on cancellation.
  • Draining before returning matters — abandoning a producer that is mid-send leaks it.
Go · 03

Interfaces

Go interfaces are satisfied implicitly: a type implements one by having the methods, with no declaration anywhere. That inverts the usual dependency direction — the consumer declares what it needs, and packages it never heard of already satisfy it.

DEFINE THE INTERFACE WHERE IT IS USED package billing (consumer) type Store interface { Get(id) (Inv, error) } small — one or two methods postgres.Repo memory.FakeRepo cached.Repo None of them import billing. None say “implements”. Having the method is the whole contract. AN INTERFACE VALUE IS TWO WORDS type value nil only when BOTH words are nil var p *MyErr = nil // value = nil var e error = p // type = *MyErr e != nil → true the type word is set, so the interface is not nil Return nil literally, never a nil typed pointer. Review heuristic: an interface with eight methods and exactly one implementation is not abstraction — it is indirection. The useful ones are small, named for behaviour (Reader, Notifier), and have at least one fake behind them.
Left is the design payoff, right is the trap. Because the consumer owns the interface, you can swap a Postgres store for a fake in a test without touching either package. Because an interface value carries a type word as well as a value word, a nil pointer stored in an interface is not nil — the most-hit gotcha in Go error handling.
L1 · Beginner

Behaviour, not hierarchy

  • No implements keyword — matching method sets is enough.
  • io.Reader / io.Writer: one method each, and half the standard library composes through them.
  • Pointer vs value receivers decide the method set: methods on *T are not in T's set. This is the usual "does not implement" compile error.
  • The empty interface any means "no information" — a smell in most signatures.
L2 · Intermediate

Seams and idiom

  • Accept interfaces, return structs — callers stay flexible, you keep freedom to add methods.
  • Typed-nil trap — a nil *T assigned to an interface is non-nil.
  • Type assertion v, ok := x.(T) and type switches for handling a closed set of cases.
  • Embedding for composition and decorators (a logging or caching wrapper that satisfies the same interface).
  • Compile-time check: var _ Store = (*Repo)(nil).
L3 · Advanced

Cost and restraint

  • Dynamic dispatch blocks inlining; boxing a value into an interface can force a heap allocation. Read -gcflags=-m before optimising.
  • Interface pollution — interfaces invented up-front, with one implementation, for a mock nobody needed. Wait for the second implementation.
  • Generics vs interfaces: generics for one algorithm over many types with no dispatch cost; interfaces for many behaviours behind one call site.
  • Adding a method is a breaking change for every external implementer — interfaces are API surface.
  • Reflection (encoding/json, ORMs) is convenient and measurably slow on hot paths.
Go · 04

Error handling

Errors in Go are ordinary values returned alongside results. There is no invisible unwinding path: every failure is a branch you can see in the diff — which is verbose, and is also why Go code is unusually easy to review for correctness.

WRAPPING BUILDS A CHAIN, NOT A STRING handler: “get invoice 42” service: “load customer” repo: “query rows” sql.ErrNoRows %w keeps the link Message reads top-down; the chain is still walkable bottom-up. errors.Is(err, sql.ErrNoRows) walks the chain, matches identity errors.As(err, &myErr) extracts the typed error and its fields The caller decides on the cause without string matching: THREE WAYS TO DESTROY IT fmt.Errorf(“load: %v”, err) %v flattens to text — errors.Is now fails log.Error(err); return err logged once per layer — six lines, one fault if err != nil { return nil } swallowed — the incident starts here Rule: %w going up, handle once at the top, log exactly where you stop returning.
Wrapping with %w is what turns error strings into a queryable structure. Each layer adds the context it uniquely has — the id, the operation — and the top of the stack still gets to ask "was this really a missing row?" and map it to a 404 rather than a 500.
// Layer boundary: add context going up, decide only at the top.
func (s *Service) Invoice(ctx context.Context, id string) (*Inv, error) {
    inv, err := s.repo.Get(ctx, id)
    if err != nil {
        // %w preserves the chain; the id is context only this layer has.
        return nil, fmt.Errorf("invoice %s: %w", id, err)
    }
    return inv, nil
}

// The one place that translates errors into a response.
func (h *Handler) Get(w http.ResponseWriter, r *http.Request) {
    inv, err := h.svc.Invoice(r.Context(), chi.URLParam(r, "id"))
    switch {
    case err == nil:
        writeJSON(w, 200, inv)
    case errors.Is(err, sql.ErrNoRows):
        http.Error(w, "not found", 404)          // expected, not an incident
    case errors.Is(err, context.DeadlineExceeded):
        http.Error(w, "timeout", 504)            // retryable, upstream slow
    default:
        h.log.Error("get invoice", "err", err)   // logged once, here
        http.Error(w, "internal", 500)
    }
}
L1 · Beginner

Errors are values

  • if err != nil immediately after the call; handle before continuing.
  • errors.New and fmt.Errorf; sentinel errors like ErrNotFound as package-level vars.
  • No exceptions. panic is for programmer bugs and unrecoverable startup failures, not for control flow.
  • defer for cleanup — and remember it runs at function exit, not scope exit.
L2 · Intermediate

Wrap, inspect, decide once

  • %w to wrap; errors.Is for identity, errors.As for type and fields.
  • Custom error types carrying structured data (field name, retry-after, status).
  • Do not log and return — pick one. Handling means the error stops here.
  • recover() only at a boundary (HTTP middleware, a worker loop) so one bad request cannot kill the process.
  • Add context that the caller cannot reconstruct; skip context it already has.
L3 · Advanced

Errors as an API

  • Taxonomy: retryable / terminal / programmer bug. It drives the retry, breaker and DLQ decisions from Pillar A.
  • context.Canceled (caller left — often not an error worth paging on) vs DeadlineExceeded (you were too slow — it is).
  • Structured errors carrying a stable code, mapped once to HTTP/gRPC status at the edge.
  • Which errors you return is public API — changing a sentinel breaks callers as surely as changing a signature.
  • Errors as telemetry: a bounded code dimension on metrics and spans, never the raw message (unbounded cardinality).
  • In concurrency: errgroup returns the first error — decide deliberately whether the others matter.
Pillar C

Cloud & infrastructure

AWS and Kubernetes are not new concepts — they are the same six foundations sold as products and expressed as YAML. A queue is SQS. A copy is a read replica. A boundary is a security group, a limit, or an availability zone. What is genuinely new is that the boundaries are now configuration, which means they can be wrong in a pull request.

Deep dive: From Account to Pod →

Cloud · 01

The AWS mental model: accounts, regions, zones

Before any service, learn the geography — because that geography is the blast radius from the fault-tolerance section, drawn on a map. Almost every AWS availability decision is really a question about which of these boxes you are willing to lose.

GLOBAL — ONE CONTROL PLANE FOR THE WHOLE ACCOUNT IAM · Route 53 · CloudFront · Organizations — not regional, so a regional outage cannot remove them REGION eu-west-1 — latency + data residency domain AZ-a AZ-b AZ-c subnet 10.0.1.0/24 subnet 10.0.2.0/24 subnet 10.0.3.0/24 node node node RDS primary RDS standby sync Lose one AZ: capacity drops by a third, RDS fails over in 60–120s, the service survives. One single-AZ component makes the whole design single-AZ. REGION us-east-1 — DR A second region doubles cost and complexity, and cross-region data is asynchronous — so failover means accepting data loss (RPO) and a recovery window (RTO). Multi-AZ is nearly free and covers most real failures. Multi-region is a business decision, not a default. The account is the outermost boundary — and the one people forget. Separate accounts per environment contain IAM mistakes, quota exhaustion and cost blowouts that no VPC rule can.
Four nested failure domains: account → region → AZ → instance. "Highly available" is meaningless until you name which of these you tolerate losing. Multi-AZ is the default answer because AZs are independent facilities on one low-latency network — you get redundancy without giving up synchronous replication.
L1 · Beginner

The map and the vocabulary

  • Account → region → AZ; an AZ is one or more real datacentres, a region is a set of them.
  • The core nouns: EC2 (a machine), S3 (objects), RDS (managed SQL), IAM (who may do what), CloudWatch (metrics and logs).
  • Shared responsibility: AWS runs the hardware and the service; you own configuration, identity, and your data.
  • Everything is an API. The console is one client — the CLI and SDK are the same calls.
L2 · Intermediate

Build it reproducibly

  • Infrastructure as code (Terraform / CDK / CloudFormation). Console changes are undocumented, unreviewable, and unrepeatable.
  • Multi-AZ by default for load balancers, node groups and databases.
  • Roles, not access keys — instance profiles and short-lived STS credentials over anything long-lived in an env var.
  • Tags drive cost allocation and ownership; untagged infra becomes nobody's.
  • Alarms on symptoms (latency, error rate, queue age), not just CPU.
L3 · Advanced

Boundaries, limits and money

  • Multi-account org with SCPs — the strongest blast-radius control AWS offers. Prod separate from everything.
  • Control plane vs data plane: during a big regional event, launching new capacity often fails while running capacity keeps serving. Design recovery that does not need the control plane.
  • Service quotas are a fault mode — hitting a limit looks exactly like an outage. Track and raise them before launch.
  • Cost is architecture: cross-AZ traffic, NAT gateway data processing and egress quietly dominate bills; VPC endpoints and AZ-aware routing are design choices.
  • RTO/RPO stated as numbers, and a restore you have actually rehearsed.
Cloud · 02

Networking and identity: the two things you can get badly wrong

The VPC decides what can reach what; IAM decides who may ask. Nearly every cloud incident that is not a capacity problem is one of these two misconfigured — usually in the permissive direction.

INBOUND PATH — EACH HOP IS A SEPARATE PERMISSION internet VPC 10.0.0.0/16 PUBLIC subnet ALB NAT via IGW · :443 PRIVATE subnet · no public IP app pods app pods SG-app allows :8080 from SG-alb only ISOLATED subnet · no route out RDS ElastiCache SG-db allows :5432 from SG-app only OUTBOUND — WHERE THE BILL AND THE LEAKS LIVE egress to 3rd-party APIs via NAT — charged per GB VPC endpoint S3 stays on the AWS network: cheaper and never public Security groups are stateful and attach to resources; NACLs are stateless and attach to subnets — a NACL needs the return-traffic rule too. Reference security groups, not CIDRs. “Allow from SG-app” keeps meaning the right thing as instances come and go; “allow from 10.0.0.0/16” quietly means “allow from everything”.
Three subnet tiers, three security groups, one direction of trust. The load balancer is the only thing the internet can reach; the app is the only thing that can reach the database. Read any VPC by asking what has a route to the internet gateway — that set should be as small as possible.
HOW EVERY AWS API CALL IS DECIDED principal asks: s3:GetObject 1. explicit Deny anywhere at all? SCP, boundary, policy 2. permitted by SCP + boundary? the org’s outer limit 3. an explicit Allow for this action? identity or resource ALLOW DENY any failed check ends here — including “nobody said anything” Default is deny. Explicit deny always wins. That asymmetry is why a broad SCP is a safety net an over-broad role cannot defeat.
Debugging "access denied" is walking this chain in order. Most confusion comes from expecting an Allow to override a Deny, or from forgetting that S3 and KMS have resource policies that must also permit the caller — cross-account access needs an Allow on both sides.
L1 · Beginner

Wires and doors

  • VPC, subnet, route table; public = has a route to the internet gateway.
  • Security group = stateful allow-list attached to a resource; there is no "deny" rule.
  • Load balancer in front, instances behind, health checks deciding who gets traffic.
  • IAM user / role / policy: a policy is JSON granting actions on resources.
L2 · Intermediate

Layouts that hold up

  • Public / private / isolated subnet tiers; databases never in a public subnet.
  • SG vs NACL — stateful per-resource vs stateless per-subnet.
  • NAT gateway for outbound, VPC endpoints to keep S3/DynamoDB traffic private and off the NAT meter.
  • ALB vs NLB (HTTP-aware and flexible vs L4, fast, static IPs); target group health checks are your readiness signal.
  • Roles assumed via STS, secrets in Secrets Manager or SSM — never in an image or a repo.
L3 · Advanced

Least privilege that survives contact

  • Policy evaluation order and why explicit deny plus SCPs are the only real guardrails.
  • IRSA / Pod Identity — per-pod AWS roles in EKS instead of one over-powered node role that every workload inherits.
  • Conditions (source VPC endpoint, MFA, tag matching) turn a coarse policy into a precise one.
  • Identity vs resource policies, cross-account trust, and confused-deputy protection.
  • Least privilege is a process: start broad in dev, tighten from CloudTrail's record of what was actually called, and use Access Analyzer to catch public exposure.
Cloud · 03

Managed services are the foundations, rented

Every service below is one of the six foundations with an SLA attached. Choosing well means knowing which concept you are buying — and which part of the concept you still own, because managed never means "no longer your problem".

Foundation AWS service What it genuinely removes What is still yours
Messaging queue SQS (standard / FIFO) Durability, scaling, redelivery, DLQ plumbing. Idempotent consumers; a visibility timeout longer than your slowest handler; draining the DLQ.
Event log Kinesis / MSK Retention, replay, ordered partitions. The partition key — and therefore your ordering guarantee and your hot-shard risk.
Event routing SNS / EventBridge Fan-out, filtering, cross-account delivery. Schema evolution and knowing who your subscribers are.
Caching ElastiCache / CloudFront Running Redis, failover, edge presence. Invalidation, key design, stampede protection, what a cold cache does to the origin.
Indexing & storage RDS / Aurora Backups, patching, Multi-AZ failover, replicas. Schema, indexes, query plans, connection limits — and 60–120s of rejected writes during failover.
Indexing at scale DynamoDB Horizontal scale with predictable single-digit-ms reads. Access patterns up front; the partition key is the index and the hot-key limit; GSIs cost writes.
Durability S3 Effectively unlimited, eleven nines of durability, versioning. Lifecycle rules, request cost at high QPS, and never making a bucket public by accident.
Fault tolerance ALB + Auto Scaling / EKS Health-checked routing, capacity replacement, multi-AZ spread. Meaningful health checks, scale-up latency, and every quota you are one traffic spike from.
L1 · Beginner

Know what each one is for

  • Files → S3. Relational data → RDS. Key-value at scale → DynamoDB. Work to do later → SQS. Hot reads → ElastiCache.
  • Compute ladder: Lambda (per request) → ECS/Fargate (containers, no nodes) → EKS (Kubernetes) → EC2 (machines).
  • Managed = AWS runs it, you configure it and you are still on call for how you use it.
L2 · Intermediate

Choose with reasons

  • RDS vs DynamoDB: flexible queries and joins, or known access patterns and horizontal scale. Picking Dynamo without knowing your queries is the classic regret.
  • SQS standard vs FIFO — throughput and at-least-once, vs ordering per group at much lower throughput.
  • SNS vs EventBridge vs Kinesis: fan-out, routed integration, or a replayable ordered stream.
  • Lambda vs a container: bursty and event-shaped, vs steady traffic, long connections, or big dependencies.
  • Autoscaling on the metric that reflects the work — queue depth or lag, not just CPU.
L3 · Advanced

The fine print that pages you

  • DynamoDB partition keys reproduce every hot-key and skew problem from Pillar A, with a hard per-partition throughput ceiling.
  • Lambda: cold starts, per-account concurrency, and one runaway function starving every other function in the account.
  • RDS connection limits vs a fleet of pods each holding a pool — a proxy or a pool budget is not optional.
  • Failover is not free: writes are rejected during it, so clients need retry with backoff and the app needs to survive a minute of read-only.
  • Lock-in and exit cost as an explicit, priced decision rather than a slogan.
  • Every managed service has a quota; quota exhaustion presents as an outage with a healthy dashboard.
Cloud · 04

Kubernetes: one idea, repeated everywhere

Kubernetes is a database of desired state plus a crowd of controllers that continuously try to make reality match it. Every object, every feature, and every surprising behaviour follows from that one sentence.

THE RECONCILIATION LOOP — RUNNING CONTINUOUSLY, NOT ON YOUR COMMAND you: kubectl apply -f app.yml API server + etcd DESIRED: replicas 3, image v2.1 controller watch → diff → act, forever watch OBSERVED 2 pods running v2.1, 1 node just died compare gap = 1 pod → create one write status back Three consequences that explain almost everything It is level-triggered, not event-triggered. The loop compares state; it does not replay your command. That is why apply is idempotent and why a missed message never leaves the cluster permanently wrong. You cannot delete a managed pod. Delete it and the controller sees a gap and makes another. To stop it, change the desire. The control plane going down does not stop your app. Running pods keep serving; you just lose the ability to change things.
Everything in Kubernetes is this loop with a different noun. Deployments reconcile ReplicaSets, ReplicaSets reconcile Pods, the HPA reconciles replica counts, cert-manager reconciles certificates — and a custom operator you write is the same loop over your own CRD.
OWNERSHIP — WHAT CREATES WHAT TRAFFIC — HOW A REQUEST FINDS A POD Deployment ReplicaSet v2.1 pod pod pod holds the pod template and the rollout strategy one per template version — the old one is kept, so rollback is instant labels: app=orders, version=v2.1 A pod is the smallest unit: one or more containers sharing a network namespace and lifetime. Pods are cattle: never patched, always replaced. Ingress / Gateway Service orders EndpointSlice host + path routing, TLS; backed by a real ALB selector: app=orders stable DNS name + virtual IP the live list of ready pod IPs, maintained by a controller matched by LABEL, never by name Consequence: a Service will happily send traffic to any pod carrying the label — including one from a different Deployment you did not mean to match.
Labels are the only glue. Nothing holds a reference to a pod; controllers run label queries and act on whatever answers. That is what makes canaries, blue/green and debugging-by-relabelling possible — and what makes a careless selector route production traffic somewhere unintended.
L1 · Beginner

The objects you touch daily

  • Pod (unit of running), Deployment (manages pods), Service (stable address), Ingress (external entry), Namespace (grouping).
  • ConfigMap and Secret for configuration; environment differences belong here, not in the image.
  • Declarative YAML plus kubectl apply; get, describe, logs, and get events are the debugging loop.
  • Containers log to stdout; the platform collects them.
L2 · Intermediate

Model your workload correctly

  • Labels and selectors as the wiring; a typo here is a silent routing bug, not an error.
  • Deployment vs StatefulSet vs DaemonSet vs Job — stateless replicas, stable identity and storage, one per node, run-to-completion.
  • Rollback is instant because the previous ReplicaSet still exists.
  • Namespaces + RBAC + ResourceQuota as the multi-team boundary.
  • On EKS: IRSA for AWS permissions per service account, and the AWS load balancer controller turning Ingress into a real ALB.
L3 · Advanced

Extend it, and know its edges

  • CRDs and operators — the same reconcile loop over your own types; the correct place for domain-specific automation.
  • Admission control (validating and mutating webhooks, policy engines) to enforce standards at write time rather than in review.
  • etcd is the system of record: its size, latency and backups are the cluster's real limits.
  • Data plane survives the control plane — design deploys and recovery so they do not require an API server that may itself be degraded.
  • Networking depth: CNI, kube-proxy vs eBPF, NetworkPolicy as the in-cluster equivalent of security groups (default-open until you write one).
Cloud · 05

Health, rollouts and shutdown

Probes are how your process tells the cluster what it can do right now, and the rolling update is entirely gated on them. Getting the three probes wrong is the most common way a healthy application becomes an outage on a busy afternoon.

THREE PROBES, THREE DIFFERENT CONSEQUENCES pod created startupProbe slow boot allowed readinessProbe may I take traffic? livenessProbe am I wedged? FAIL → out of endpoints no traffic, still running, rejoins when it recovers FAIL → container killed restarted from scratch — destructive, use sparingly The classic self-inflicted outage A liveness probe that checks the database. DB blips → every probe fails → every pod restarts at once → cold caches and a connection stampede when the DB returns. liveness: is the process alive. Nothing else. ROLLING UPDATE — READINESS IS THE GATE v2.0 v2.1 maxSurge: 1 → start one new pod wait for READY → only then remove an old one maxUnavailable: 0 keeps full capacity throughout. Without a readiness probe, Kubernetes assumes a started container is ready — and a rollout becomes a synchronised outage at the speed of your image pull. SHUTDOWN — IN THAT ORDER pod marked Terminating, removed from endpoints preStop sleep — lets that removal actually propagate SIGTERM → your Go code cancels ctx and drains SIGKILL after terminationGracePeriodSeconds Skip the preStop pause and you drop in-flight requests on every deploy.
Readiness removes traffic; liveness destroys the container. They are not two names for the same check. Readiness may depend on downstreams — carefully, because if every pod marks itself unready during a dependency blip you have converted a partial degradation into a total outage. Liveness should only ever answer "is this process still functioning".
Cloud · 06

Resources, scheduling and scaling

Requests are a promise the scheduler makes to your pod; limits are a promise your pod makes to the node. CPU and memory break that promise in completely different ways, and knowing which is which is most of Kubernetes performance debugging.

REQUESTS BUY YOU A SEAT. LIMITS DECIDE WHAT HAPPENS WHEN YOU STAND UP. node: 4 CPU req 1 req 1.5 req 0.75 free The scheduler only ever looks at requests — it places pods by reservation, not by actual usage. CPU over limit → THROTTLED, not killed latency spikes with idle cores and no error in your logs memory over limit → OOMKilled instantly no grace period, no drain, exit code 137 CPU is compressible, memory is not — which is why memory limits must be near-real, and why many teams leave CPU limits off. QoS — WHO GETS EVICTED FIRST WHEN THE NODE IS SQUEEZED BestEffort Burstable Guaranteed no requests set requests == limits · safest THE SCALING LOOP — CLOSED WITH PILLAR A queue lag rises HPA adds pods no room on nodes Karpenter adds node Scale on the metric that is the backlog — consumer lag, queue depth, in-flight requests. CPU is a lagging proxy that stays flat while an I/O-bound worker falls behind. A new node takes 60–120s to arrive. Scaling has latency, and traffic does not wait Metric window + HPA interval + pod start + node provision can total two or three minutes. Keep headroom, pre-scale for known peaks, and shed load for the gap.
Most "Kubernetes is slow" reports are one of these three boxes. A pod at its CPU limit is throttled while the node sits idle; a pod slightly over its memory limit dies without warning; a BestEffort pod is evicted first when a neighbour grows. Right-sizing requests is not cost hygiene — it is latency work.
L1 · Beginner

Ask for what you need

  • Requests = reserved for scheduling. Limits = the ceiling.
  • Always set both for memory; set CPU requests at minimum.
  • kubectl top, and reading OOMKilled / CrashLoopBackOff in describe.
  • More replicas is horizontal scaling; a bigger pod is vertical. Prefer horizontal for stateless work.
L2 · Intermediate

Place and scale deliberately

  • CPU throttling vs OOMKill — two entirely different symptoms; check throttling metrics before blaming the code.
  • QoS classes decide eviction order under node pressure.
  • topologySpreadConstraints across AZs and nodes, so three replicas are not three pods on one machine in one AZ.
  • PodDisruptionBudget so node upgrades and scale-downs cannot drain your service to zero.
  • HPA on a custom or external metric; Cluster Autoscaler / Karpenter for nodes.
L3 · Advanced

Density, tails and money

  • Should you set CPU limits at all? A real trade: limits give predictable neighbours, and cause p99 throttling even on an idle node. Decide per workload, with data.
  • Go on Kubernetes: without GOMAXPROCS awareness the runtime sizes itself to the node's cores, not your CPU quota — instant throttling under load.
  • Scaling latency budget end to end; combine headroom, pre-scaling and load shedding rather than trusting the autoscaler alone.
  • Spot capacity with interruption handling, plus PDBs and spread — large savings, only if the workload is genuinely disposable.
  • Bin packing vs blast radius: denser nodes are cheaper and make each node failure hurt more.
Synthesis

One level up: it is all the same four moves

Studied separately, these look like sixteen topics. Studied together, they are four moves applied to different resources — Go's features are the in-process version, and AWS and Kubernetes are the rented version. That collapse is what "deep foundations" means in practice: you meet an unfamiliar system and already know which four questions to ask.

THE MOVE AT SYSTEM SCALE INSIDE ONE GO PROCESS ON AWS / KUBERNETES THE PRICE YOU PAY Add a buffer decouple in time message queues, event log, write-behind, outbox buffered channels, worker pools SQS · Kinesis · EventBridge pod queue depth → HPA latency, and a backlog to own Add a copy trade memory for work caches, indexes, replicas, read models / CQRS value semantics, sync.Map, per-goroutine state ElastiCache · CloudFront read replicas · DynamoDB GSI staleness, and invalidation Add an order make concurrency safe partition keys, single writer, transactions, idempotency mutex, atomics, unbuffered channel handoff SQS FIFO · Dynamo conditional writes · StatefulSet identity contention — the scaling ceiling Add a boundary contain the damage timeouts, breakers, bulkheads, cells, rate limits context deadlines, errors as values, recover at the edge accounts · AZs · VPC · SG · IAM namespaces · limits · PDB complexity, and false rejects Read any row across: it is one idea wearing four uniforms. Learning the row beats learning the cells.
The same four moves, at three scales. A goroutine reading a buffered channel, a consumer group reading a Kinesis shard, and an HPA reading queue depth are the identical shape at different radii. AWS and Kubernetes do not add concepts here — they give the moves a price tag, an SLA, and a YAML file.

Everything at once: one request through one service

Here is a single order-placement path with every concept on this page marked where it actually appears, and the infrastructure it lands on written underneath. If you can narrate this diagram out loud — including what happens when each box is slow, when each box is dead, and which AZ each one lives in — you are answering the job description.

SYNCHRONOUS — INSIDE THE USER’S 300ms ASYNCHRONOUS — EVENTUALLY CONSISTENT client HTTP handler ctx, 300ms deadline recover() middleware goroutine per request cache catalog + jitter cache-aside + singleflight ONE TX orders row + index outbox row indexing + no dual write card auth (sync) timeout 200ms breaker + idem key must be sync: user needs the answer breaker OPEN → queue for review graceful degradation, not a 500 201 Created, order accepted outbox relay at-least-once tails committed rows event log key = order_id 12 partitions ordering per order settlement worker inventory worker search projection each: errgroup of 8, idempotent by event id DLQ alert on lag, age, DLQ depth saga: refund compensate The dashed line is the real design decision: what must be true before we answer the user, and what is allowed to become true afterwards. AND WHERE IT ALL ACTUALLY RUNS handler + workers Deployment, 6 pods, spread over 3 AZs cache / DB ElastiCache + RDS Multi-AZ, isolated subnet event log + DLQ Kinesis or MSK, SQS DLQ, IRSA for access scaling + entry ALB via Ingress, HPA on consumer lag
One diagram, every concept. Concurrency is the handler and the worker pools; caching is the catalog read; indexing is the write; the outbox and log are the event backbone; the partition key is the ordering guarantee; timeouts, breakers, DLQs and the saga are the fault tolerance. In Go this is roughly 600 lines and no framework — and the bottom band is the only part that changes if you move it between EKS, ECS and a rack of your own machines.

The four questions to ask about any box on any diagram

1. Where is the queue, and is it bounded?  2. What is the contract, and who breaks if it changes?  3. What happens when this is slow — who waits, and for how long?  4. What happens when this is dead — do we degrade, queue, or fail?  Most senior interviews and most postmortems are these four questions in a trench coat.

Self-check

The whole ladder on one page

One row per concept. Read across to find where you actually are, and read the L3 column to find the sentence you cannot yet say without hedging — that is the next thing to learn.

Concept L1 — can name it L2 — can operate it L3 — owns the tradeoff
Concurrency Races exist; a mutex protects a critical section. Bounded pools, deadlock avoidance, cancellation, visibility. Contention is the scaling ceiling; partitioning beats locking.
Event-driven Events are past-tense facts; publishers don't know subscribers. Idempotent consumers, per-key ordering, outbox over dual write. Sagas, replay safety, schema evolution as a forever commitment.
Caching TTL, eviction, hit ratio, which layer to put it in. Cache-aside, key versioning, invalidate on write, negative caching. Stampede, cold start, hot keys — the cache is also a new outage.
Indexing Turns a scan into a lookup; costs write throughput. Composite column order, selectivity, covering indexes, reading plans. LSM vs B-tree amplification, plan instability, shard key over index.
Message queues Producer, broker, consumer, ack, retry. At-least-once + idempotency, consumer groups, DLQ, lag alerts. Log vs broker, hot partitions, poison messages, backpressure.
Fault tolerance Always time out; redundancy; degrade rather than fail. Backoff + jitter, breakers, bulkheads, deadline propagation. Metastable failure, blast radius, gray failure, error budgets.
Go · goroutines go f(), WaitGroup, -race. Context everywhere, bounded pools, errgroup, leak review. Scheduler behaviour, GOMAXPROCS in containers, pprof/goleak.
Go · channels Send, receive, close, range, buffered vs not. select with ctx.Done, pipelines, semaphores, ownership rules. Happens-before, when a mutex is the better tool, deadlock shapes.
Go · interfaces Implicit satisfaction; method sets and receivers. Accept interfaces/return structs, typed-nil trap, test seams. Dispatch and allocation cost, interface pollution, generics vs interfaces.
Go · errors Errors are values; check immediately; no exceptions. %w, errors.Is/As, handle once, recover at boundaries. Retryable vs terminal taxonomy; errors as public API and telemetry.
AWS · the map Account, region, AZ; the core services; shared responsibility. Multi-AZ by default, IaC, roles over keys, alarms on symptoms. Multi-account blast radius, control vs data plane, quotas, RTO/RPO, cost as design.
AWS · network & IAM VPC, subnets, security groups, load balancer, policies. Public/private/isolated tiers, SG vs NACL, VPC endpoints, STS roles. Policy evaluation order, IRSA, conditions, least privilege from CloudTrail.
AWS · managed services Which service solves which problem. RDS vs DynamoDB, SQS vs FIFO vs Kinesis, Lambda vs containers. Partition-key limits, failover windows, connection ceilings, quota exhaustion.
K8s · model Pod, Deployment, Service, Ingress, ConfigMap; apply and describe. Labels as wiring, workload types, instant rollback, RBAC and quotas. CRDs and operators, admission control, etcd limits, data plane independence.
K8s · running it Requests and limits; reading OOMKilled and CrashLoopBackOff. Three probes, rolling updates, graceful shutdown, PDB, spread, HPA. Throttling vs OOM, cascading readiness, GOMAXPROCS in a quota, scaling latency.

If you only take three things into the interview

Backpressure — every buffer must be bounded, and a full buffer must slow the producer rather than silently grow. Idempotency — it is what makes retries, queues, replays, outboxes and every at-least-once managed service safe, and it is a design decision, not a library. Blast radius — goroutine, pod, node, AZ, region, account: name the boundary you are willing to lose, and check that no single component quietly spans all of them.