hardNewAnthropicMeta

Design disaster recovery for a three-month training run: checkpoint replication, cluster failover, and the RTO you can promise.

A three-month run on 8,000 GPUs is a large bet against everything that can happen to one building. The recovery point and the recovery time as numbers, the checkpoint replication that sets the first, the second cluster and its warm state that set the second, and the drills that make the numbers true.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

📐 AI Systems Design

Multi-Region Serving and FailoverRunning inference in more than one region buys latency for distant users and survival when a region fails, and it costs a second fleet that must be capable of absorbing the first one's traffic. The design turns on three decisions: whether regions are active-active or active-passive, what state has to cross regions and what deliberately does not, and how much headroom each region carries so a failover does not simply move the outage.

Core

🔌 Networking & StorageSign in

Parallel Filesystems vs Object StorageA training cluster's storage has two very different jobs: stream terabytes of training data to thousands of GPUs at a steady rate, and absorb a multi-terabyte checkpoint burst every few minutes. Parallel filesystems (Lustre, GPFS, WEKA, VAST, FSx) give POSIX semantics and hundreds of GB/s of aggregate throughput; object storage (S3 and its equivalents) gives durability and cost at a fraction of the price with high first-byte latency. Almost every real cluster uses both, and the interview question is which job goes where and how big each tier has to be.

Core

🧮 Napkin Math & CapacitySign in

GPU-Hours and Time to TrainThe fleet equation turns a training run's FLOPs into a schedule: time = 6ND divided by (GPUs times peak FLOPS times MFU). Every term is a stated assumption, and the interviewer grades the assumptions rather than the digits: which peak, which MFU, and what happens to the answer when MFU falls from 40% to 30%. This page works three runs end to end (an 8B, a 70B and a 405B), inverts the equation for the GPU count a deadline needs, and shows the sensitivity that separates a considered estimate from a lucky one.

Foundational

🕸️ Distributed Training

Data Parallelism and DDPData parallelism gives every GPU a full copy of the model, feeds each a different slice of the batch, and averages the gradients with an all-reduce so every replica takes the same optimizer step. It is the first parallelism every training job uses, and the tokens-per-GPU arithmetic behind it decides whether the communication hides behind the backward pass or dominates the step.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign token-level billing and metering for an LLM API: count at the engine, aggregate, reconcile, and survive the failure modes.Next in this trackDesign prefill and decode disaggregation for a 70B serving fleet: pool sizes, KV transfer, routing, and when to turn it on.Next in this trackDesign an internal LLM gateway for a company: routing across providers and models, caching, fallbacks, and cost controls.

FEDITOR'S NOTE

Scored on RPO and RTO as derived numbers, on replication that keeps up with the checkpoint cadence, on a failover cluster whose readiness is measured in hours not weeks, and on drills as part of the plan rather than a wish.

DISCUSSION · 0

No comments yet — be the first to share your approach.