hardNewOpenAIAnthropicGoogle

Design a multi-region inference deployment: capacity per region, routing, failover, and getting the weights everywhere.

Two regions at 60% is not one region with a spare: the question is what happens in the 30 seconds after a region drops. Capacity sized for N-1 with the arithmetic, latency routing with a residency override, failover that does not stampede, weights warm everywhere in advance, and the state that must not cross a border.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

📐 AI Systems Design

Multi-Region Serving and FailoverRunning inference in more than one region buys latency for distant users and survival when a region fails, and it costs a second fleet that must be capable of absorbing the first one's traffic. The design turns on three decisions: whether regions are active-active or active-passive, what state has to cross regions and what deliberately does not, and how much headroom each region carries so a failover does not simply move the outage.

Advanced

📐 AI Systems Design🔒 Premium

Request Routing and Load Balancing for LLMsA load balancer for stateless web services spreads requests evenly and is done. A router for LLM replicas has two things a web balancer never had to think about: each replica holds a cache (the KV pages of recent prefixes) that makes some replicas far cheaper than others for a given request, and each request costs a wildly different amount, so counting connections is meaningless. This page builds the router that handles both: prefix-aware placement with load-aware fallback, cost-aware queue estimates, session affinity, and the failure handling when a replica restarts and its cache is gone.

Foundational

💻 Coding for Infra

Consistent Hashing and ShardingSplitting work across N servers with a modulo of N moves almost everything when N changes, which for a cache means throwing away almost all of it. Consistent hashing places servers and keys on a ring so adding or removing one moves only its share, and virtual nodes fix the imbalance a small ring otherwise has. In LLM serving the same structure routes requests by prompt prefix so a conversation reaches the replica already holding its cache.

Core

📐 AI Systems DesignSign in

Designing for Latency SLOsA latency objective is met or missed by the sum of a chain of delays, and the way to design for it is to write the chain down with a number on every link, find the links that dominate at the tail, and attack those. For an LLM request the chain is network, gateway, router, queue, prefill, then the decode loop, and the tail is shaped by queueing and by the size of the batch the request lands in. This page decomposes a 500 ms time-to-first-token budget link by link, derives how queueing turns a comfortable median into a broken p99, and gives the design moves (admission control, chunked prefill, priority lanes, hedging) that hold it.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the playground behind an LLM API: streaming, sessions, quotas and abuse, for a million free-tier users.Next in this trackDesign the checkpoint store for a lab running several large training jobs: write bursts, retention, resharding and lineage.Next in this trackDesign observability for an inference fleet: per-request spans, GPU metrics, and cost attribution per tenant.

FEDITOR'S NOTE

Scored on N-1 capacity arithmetic per model, on latency-based routing with data-residency as a hard constraint, on failover that shifts traffic gradually against cold caches, and on treating weights and KV state as region-local.

DISCUSSION · 0

No comments yet — be the first to share your approach.