Advanced

Checkpointing and Resumption at Scale

A training checkpoint at frontier scale is terabytes of sharded optimizer state that must be written often enough to bound lost work and fast enough not to stall the job. The interval is a formula in the failure rate and the write cost, and asynchronous sharded writes are what turn it from a 15% tax into a 3% one.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Elastic and Fault-Tolerant Training ZeRO and FSDP Checkpoint I/O Training Uptime and Interruption Statistics Parallel Filesystems vs Object Storage Spot, Preemption and Capacity Strategies

LESSONS THAT TEACH THIS

Training at ScaleCheckpoint cadence is an optimisation against your own failure rate16 min→Training at ScaleCapstone: plan a large run end to end and defend the schedule300 min→Fleet and ReliabilityQuota is a policy question that engineers keep being asked to answer15 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismAt 16,000 GPUs something fails every few hours. How do you choose the checkpoint interval, and what does the write have to look like?→Distributed Training & ParallelismDesign a checkpoint format for thousands of GPUs: no gather on write, resumable at a different world size, no stall.→Distributed Training & ParallelismYour 4,096-GPU run loses about 2% of every day to restarts. Fix it, and tell me where the floor is.→Napkin Math, Cost & CapacityEstimate how long it takes to write a checkpoint for a 405B training run→GPU Fleet Reliability & ObservabilityAt 16,384 GPUs something fails every three hours. How often should you checkpoint, and what goodput does that leave?→GPU Fleet Reliability & ObservabilityHow do GPUs actually fail at fleet scale, how often, and which failures should the platform expect to handle every day?→