At 16,000 GPUs something fails every few hours. How do you choose the checkpoint interval, and what does the write have to look like?

Meta's Llama 3 report counted 419 unexpected interruptions in 54 days on 16,384 GPUs, one every three hours. The failure-rate arithmetic, the loss as a function of the interval, the square-root formula that minimizes it, and why the write must be sharded and asynchronous first.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🕸️ Distributed Training🔒 Premium

Checkpointing and Resumption at ScaleA training checkpoint at frontier scale is terabytes of sharded optimizer state that must be written often enough to bound lost work and fast enough not to stall the job. The interval is a formula in the failure rate and the write cost, and asynchronous sharded writes are what turn it from a 15% tax into a 3% one.

Advanced

🔌 Networking & Storage🔒 Premium

Checkpoint I/OA checkpoint of a 70B model's training state is 1.13 TB, and a 405B model's is 6.5 TB. Written synchronously every half hour, it stalls thousands of GPUs for as long as the slowest rank takes to reach the disk. The design that scales writes each rank's shard in parallel, copies it off the GPU to host memory first so the run resumes in seconds, and drains it to durable storage in the background. The arithmetic is bytes per rank against the bandwidth of each hop, and the goal is a checkpoint that costs the run under 1% of its time.

Foundational

🩺 Fleet Reliability & Observability

Training Uptime and Interruption StatisticsA frontier training run is a months-long job on tens of thousands of parts, each of which fails rarely, so the run as a whole fails constantly. The numbers that describe this are simple and worth deriving from scratch: a per-GPU failure rate, multiplied by the fleet, gives the interval between stops; the checkpoint interval and the restart time give the work lost per stop; together they give effective training time, which is the fraction of wall-clock the run is actually learning. This page derives all of it from the one public dataset that reports it and shows how to size checkpointing so the run stays above 90%.

Foundational

🕸️ Distributed Training

Data Parallelism and DDPData parallelism gives every GPU a full copy of the model, feeds each a different slice of the batch, and averages the gradients with an all-reduce so every replica takes the same optimizer step. It is the first parallelism every training job uses, and the tokens-per-GPU arithmetic behind it decides whether the communication hides behind the backward pass or dominates the step.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign a training system that survives losing a node without a human in the loop. What does elasticity cost you?Next in this trackOne GPU out of 16,000 is 15% slow and the whole run is 15% slow. How do you find it, and what is usually wrong with it?Next in this trackYou see 'Watchdog caught collective operation timeout' in the logs. What happened, and how do you find which rank caused it?

FEDITOR'S NOTE

Scored on deriving the per-GPU failure rate from a public record, on writing the loss function of the interval and minimizing it, and on knowing that the checkpoint cost, not the storage, sets the floor on the interval.

DISCUSSION · 0

No comments yet — be the first to share your approach.