Your 4,096-GPU run loses about 2% of every day to restarts. Fix it, and tell me where the floor is.

Two restarts a day at fifteen minutes each is the 2%. The loss decomposed into detection, rescheduling, reload and rewound work, the lever on each, the in-memory checkpoint that makes the interval a minute, the square-root pareto of interval against write cost, and the residual that only fewer failures can remove.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🕸️ Distributed Training🔒 Premium

Elastic and Fault-Tolerant TrainingAt 16,384 GPUs something fails every few hours, so a training job is a loop of detect, isolate, replace and resume rather than one process that runs for months. The Llama 3 record of 466 interruptions in 54 days is the reference point, and the design goal is keeping effective training time above 90% by making each event cost minutes.

Advanced

🕸️ Distributed Training🔒 Premium

Checkpointing and Resumption at ScaleA training checkpoint at frontier scale is terabytes of sharded optimizer state that must be written often enough to bound lost work and fast enough not to stall the job. The interval is a formula in the failure rate and the write cost, and asynchronous sharded writes are what turn it from a 15% tax into a 3% one.

Foundational

🩺 Fleet Reliability & Observability

Training Uptime and Interruption StatisticsA frontier training run is a months-long job on tens of thousands of parts, each of which fails rarely, so the run as a whole fails constantly. The numbers that describe this are simple and worth deriving from scratch: a per-GPU failure rate, multiplied by the fleet, gives the interval between stops; the checkpoint interval and the restart time give the work lost per stop; together they give effective training time, which is the fraction of wall-clock the run is actually learning. This page derives all of it from the one public dataset that reports it and shows how to size checkpointing so the run stays above 90%.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Incident Response for GPU FleetsAn incident on a GPU fleet is a training run that stopped, a serving endpoint burning its error budget, or a fleet-wide symptom nobody has explained yet. The response has a shape: detect, stabilize, diagnose, repair, return through the gate, write it up. The stabilizing move (drain the node, restart from checkpoint, or shift traffic) comes before the diagnosis, because a frontier run loses more per minute than any investigation is worth. This page gives the triage order, the 3am decision tree, the spare-capacity arithmetic behind drain-and-replace, and what a fleet postmortem has to contain.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackPlan a 10-trillion-token pre-training run end to end: compute, fleet, layout, data, checkpoints and a schedule with a failure budget.Closest to this topicDesign a training system that survives losing a node without a human in the loop. What does elasticity cost you?Closest to this topicA GPU in a running 512-GPU training job is throwing errors. How do you get it out of the job without losing the run?

FEDITOR'S NOTE

Scored on decomposing the loss per failure into named stages with minutes on each, on attacking the largest term first, on knowing that the interval and the write cost trade against each other, and on the residual set by the failure rate itself.

DISCUSSION · 0

No comments yet — be the first to share your approach.