Multi-Region Serving and Failover
Running inference in more than one region buys latency for distant users and survival when a region fails, and it costs a second fleet that must be capable of absorbing the first one's traffic. The design turns on three decisions: whether regions are active-active or active-passive, what state has to cross regions and what deliberately does not, and how much headroom each region carries so a failover does not simply move the outage.
TL;DR: Decide the three things in order. Active-active means every region serves traffic and a failure sheds its share onto the others, which needs each region to carry enough spare capacity to absorb it; active-passive keeps a second fleet warm and mostly idle, which is simpler and more expensive per served request. Then decide what crosses regions: model weights and configuration must be replicated, request routing state should not, and the KV cache should never, because a conversation that moves region loses its cache and pays a full prefill. Then size the headroom, because two regions each running at 80 percent cannot absorb each other. With R regions each at utilization u, surviving one failure requires u below (R-1)/R, so two regions must each stay under 50 percent and four under 75, which is the arithmetic that makes two-region active-active expensive and four-region active-active reasonable.
The headroom arithmetic, which decides the shape
R regions, each at utilization u of its own capacity, one region fails
the failed region's load is redistributed across the remaining R - 1
each survivor now carries u + u/(R-1) = u x R/(R-1)
surviving requires u x R/(R-1) <= 1, so u <= (R-1)/R
R = 2: u <= 0.50
R = 3: u <= 0.67
R = 4: u <= 0.75
R = 6: u <= 0.83
what that costs
two-region active-active pays for twice the capacity it uses, since each region idles half
four regions pay for 1.33 times, which is why large active-active deployments have several
regions rather than two
sanity: this is the same N+1 argument as replicas within a region, applied one level up, and
it is the reason a two-region design is often better run active-passive than
active-active
What crosses a region and what does not
| State | Crosses? | Why |
|---|---|---|
| Model weights and revisions | Yes, replicated ahead of time | A cold pull across regions is hours; mirror per region |
| Configuration and routing policy | Yes | Regions must agree on which model version is current |
| Request routing decisions | No | Each region routes within itself; cross-region routing adds a wide-area hop per request |
| KV cache and prefix cache | No | Moving a conversation's cache costs more than re-prefilling and the transfer crosses a wide-area link |
| Usage accounting and audit | Yes, asynchronously | It must be complete eventually, not instantly |
| Quotas and rate limits | Usually per region, with a global cap reconciled asynchronously | A synchronous global limiter puts a wide-area round trip in the request path |
why the cache must not cross
a conversation pinned to region A has its prefix cached there
moving it to region B means a full prefill of the shared head
at a 2,000-token head on a 70B model: 2 x 70e9 x 2,000 = 2.8e14 FLOPs of work repeated
and the alternative, shipping the KV, is hundreds of megabytes over a wide-area link
sanity: session affinity to a region is therefore a correctness-adjacent property rather than
an optimization, and a failover deliberately accepts losing it
The failover itself
Two behaviours have to be designed rather than assumed. Hysteresis, because a region that flaps in and out moves traffic repeatedly and each move costs the cache affinity of every session it carries. And graceful degradation, because the honest outcome when survivors cannot hold the objective is shedding or queueing with a stated wait rather than accepting everything and missing the target for everyone.
When one region is enough
Multi-region is frequently premature. It doubles the operational surface, requires the weight replication and configuration discipline above, and buys nothing if the product's users are concentrated in one place and the availability target is met by replica-level redundancy within a region. The questions that decide it are where the users are, what the availability target actually is in numbers, and whether a regional failure is a scenario the business has priced. Building two regions to raise availability that was already sufficient is a common and expensive way to make a platform harder to operate.
What interviewers are listening for
The headroom arithmetic, because it is the thing that makes multi-region concrete and almost nobody produces it unprompted. After that, what deliberately does not cross, since a candidate who proposes replicating the KV cache has not thought about the cost. The third signal is naming the degraded mode: what happens when the survivors cannot hold the objective, which is a product decision that has to be made before the incident rather than during it.
Key takeaways
- Surviving one region failure requires each region below (R-1)/R utilization: 50 percent at two regions, 75 at four.
- Replicate weights and configuration ahead of time; never move KV cache or route requests across regions.
- Session affinity to a region is what keeps prefix caches useful, and a failover accepts losing it.
- Design hysteresis into the health signal, because a flapping region costs the cache affinity of every session it moves.
- Decide the degraded mode in advance: shed or queue with a stated wait rather than missing the objective for everyone.