Advanced

Elastic and Fault-Tolerant Training

At 16,384 GPUs something fails every few hours, so a training job is a loop of detect, isolate, replace and resume rather than one process that runs for months. The Llama 3 record of 466 interruptions in 54 days is the reference point, and the design goal is keeping effective training time above 90% by making each event cost minutes.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Checkpointing and Resumption at Scale Stragglers and Hangs GPU Failure Modes and XID Errors Training Uptime and Interruption Statistics Node Health Checks and Burn-In Collective Communication Primitives

LESSONS THAT TEACH THIS

Training at ScaleCheckpoint cadence is an optimisation against your own failure rate16 min→Training at ScaleAt ten thousand devices, failure is a line in the schedule16 min→Training at ScaleA loss curve that goes wrong is a systems bug until proven otherwise16 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismDesign a training system that survives losing a node without a human in the loop. What does elasticity cost you?→Kubernetes, Slurm & GPU SchedulingA GPU in a running 512-GPU training job is throwing errors. How do you get it out of the job without losing the run?→Distributed Training & ParallelismYour 4,096-GPU run loses about 2% of every day to restarts. Fix it, and tell me where the floor is.→Distributed Training & ParallelismIn data-parallel training, what actually gets communicated between GPUs, and how much is it per step?→Distributed Training & ParallelismCompare data, tensor and pipeline parallelism. What does each one shard, what does each one communicate, and where does each one live?→Distributed Training & ParallelismWhat is MFU, how do you compute it from a running job, and what counts as a good number?→