A GPU in a running 512-GPU training job is throwing errors. How do you get it out of the job without losing the run?

You cannot remove one rank from a synchronous job; you replace the node and restart from checkpoint, and the whole craft is making that take three minutes instead of thirty. The detection signals, the drain-restart-quarantine sequence, the spare-pool arithmetic, and where elastic training changes the answer.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🕸️ Distributed Training🔒 Premium

Elastic and Fault-Tolerant TrainingAt 16,384 GPUs something fails every few hours, so a training job is a loop of detect, isolate, replace and resume rather than one process that runs for months. The Llama 3 record of 466 interruptions in 54 days is the reference point, and the design goal is keeping effective training time above 90% by making each event cost minutes.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Incident Response for GPU FleetsAn incident on a GPU fleet is a training run that stopped, a serving endpoint burning its error budget, or a fleet-wide symptom nobody has explained yet. The response has a shape: detect, stabilize, diagnose, repair, return through the gate, write it up. The stabilizing move (drain the node, restart from checkpoint, or shift traffic) comes before the diagnosis, because a frontier run loses more per minute than any investigation is worth. This page gives the triage order, the 3am decision tree, the spare-capacity arithmetic behind drain-and-replace, and what a fleet postmortem has to contain.

Advanced

🔌 Networking & Storage🔒 Premium

Checkpoint I/OA checkpoint of a 70B model's training state is 1.13 TB, and a 405B model's is 6.5 TB. Written synchronously every half hour, it stalls thousands of GPUs for as long as the slowest rank takes to reach the disk. The design that scales writes each rank's shard in parallel, copies it off the GPU to host memory first so the run resumes in seconds, and drains it to durable storage in the background. The arithmetic is bytes per rank against the bandwidth of each hop, and the goal is a checkpoint that costs the run under 1% of its time.

Foundational

🗂️ Scheduling & Orchestration

Node Lifecycle: Drain, Upgrade and ReturnA node moves through a fixed cycle between provisioning and decommissioning, and most fleet operations are one lap around it: cordon so nothing new lands, drain so running work finishes or moves, act, validate, then return to the pool. The wall-clock cost of a fleet-wide change is dominated by draining rather than by the change itself, which makes the plan a scheduling document rather than a technical one.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackExplain how a Slurm job actually gets its GPUs: GRES, partitions, cgroups, and the sbatch that lands 8 GPUs on one node.Next in this trackWe want to run LLM inference on Kubernetes. What does the deployment actually need beyond a Deployment with a GPU limit?Next in this trackDesign the scheduling and isolation for a multi-tenant fine-tuning service: hundreds of customers, a few base models, shared GPUs.

FEDITOR'S NOTE

Scored on saying plainly that a synchronous gang cannot lose a rank (restart is the mechanism), on the automated sequence with a time per step, on the spare-pool derivation, and on knowing when elastic frameworks change it.

DISCUSSION · 0

No comments yet — be the first to share your approach.