Training Uptime and Interruption Statistics
A frontier training run is a months-long job on tens of thousands of parts, each of which fails rarely, so the run as a whole fails constantly. The numbers that describe this are simple and worth deriving from scratch: a per-GPU failure rate, multiplied by the fleet, gives the interval between stops; the checkpoint interval and the restart time give the work lost per stop; together they give effective training time, which is the fraction of wall-clock the run is actually learning. This page derives all of it from the one public dataset that reports it and shows how to size checkpointing so the run stays above 90%.
TL;DR: Meta's Llama 3 run: 16,384 H100s, 54 days, 466 interruptions of which 419 were unexpected, about 78% hardware, and yet more than 90% effective training time. Derive it: 419 stops in 54 days is one every 3.1 hours; with a checkpoint interval of T and a restart cost of R, each stop loses about T/2 + R; at T = 30 min and R = 10 min that is 25 minutes per stop, 25 ÷ 186 minutes ≈ 13% lost, so hitting 90% needs T near 15 minutes or R near 5. The Young-Daly rule gives the checkpoint interval that minimizes loss: T_opt ≈ √(2 × C × M), with C the checkpoint cost and M the mean time between failures; at C = 1 min and M = 3.1 h it is about 19 minutes. At 100,000 GPUs the interval between stops is 30 minutes and the same arithmetic forces asynchronous checkpoints and automatic restart, or the run makes no progress.
The dataset
The Llama 3 paper is the only frontier-scale run whose interruption statistics were published in detail, which makes it the reference every interviewer and candidate shares.
| Quantity | Value |
|---|---|
| GPUs | 16,384 H100 |
| duration of the reported window | 54 days |
| interruptions, total | 466 |
| planned (maintenance, config changes, dataset updates) | 47 |
| unexpected | 419 |
| unexpected attributed to hardware (confirmed or suspected) | about 78% |
| largest categories | faulty GPU ~30%, GPU HBM ~17%, software bugs ~13%, network switch and cable ~8%, host maintenance ~8%, GPU SRAM ~5%, GPU system processor ~4% |
| requiring significant manual intervention | 3 |
| effective training time | above 90% |
Two things stand out. Failures are dominated by the GPU and its memory rather than by the network, which surprises people who expect the fabric to be the weak point. And only three of 419 needed a human, meaning the run survived on automation: detection, checkpoint restore, and restart on a replacement node.
The derivation, step by step
Step one, the fleet failure interval.
unexpected stops per day = 419 ÷ 54 ≈ 7.76
mean time between stops, fleet-wide: M = 24 ÷ 7.76 ≈ 3.1 h ≈ 186 min
Step two, the per-device rate, so the number transfers to a different fleet.
GPU-hours in the window = 16,384 × 54 × 24 ≈ 21.2 million
failures per GPU-hour ≈ 419 ÷ 21.2M ≈ 2.0 × 10⁻⁵ (every stop charged to a GPU, an upper bound)
per-GPU MTBF ≈ 1 ÷ 2.0 × 10⁻⁵ ≈ 50,600 h ≈ 5.8 years
annualized failure rate per GPU ≈ 8,760 ÷ 50,600 ≈ 17% per year (with the same caveat)
Step three, transfer to another fleet size.
M(N) ≈ 50,600 h ÷ N
N = 512: ≈ 99 h (four days)
N = 4,096: ≈ 12 h
N = 16,384: ≈ 3.1 h
N = 100,000: ≈ 30 min
Step four, the cost of each stop. A checkpoint every T minutes loses on average T/2 minutes of progress when the stop lands at a random point in the interval, plus the restart cost R (detect the failure, drain, reschedule on a spare, load the checkpoint, rebuild the collective, warm up).
loss per stop ≈ T/2 + R
lost fraction ≈ (T/2 + R) ÷ M
effective training time ≈ 1 − lost fraction − checkpoint overhead, where checkpoint overhead ≈ C ÷ T
(C = the time the run is stalled while a checkpoint is taken; near zero with async checkpoints)
Step five, put Llama 3's numbers in.
M = 186 min
T = 30 min, R = 10 min, C = 1 min: loss = (15 + 10) ÷ 186 ≈ 13.4%; overhead = 1 ÷ 30 ≈ 3.3% → ~83% effective
T = 15 min, R = 5 min, C = 1 min: loss = (7.5 + 5) ÷ 186 ≈ 6.7%; overhead ≈ 6.7% → ~87% effective
T = 15 min, R = 5 min, C ≈ 0 (async): loss ≈ 6.7%; overhead ≈ 0 → ~93% effective
sanity: reported effective training time was above 90%, which is consistent with a checkpoint interval
in the tens of minutes, a restart in single-digit minutes, and checkpoints that barely stall the run.
The optimal checkpoint interval
There is a closed form for the T that minimizes the sum of the two losses (checkpoint overhead C/T, which falls as T grows, and expected rework T/2 ÷ M, which rises). Setting the derivative to zero gives the Young-Daly approximation:
T_opt ≈ √(2 × C × M)
C = 1 min, M = 186 min: T_opt ≈ √(372) ≈ 19 min
C = 5 min, M = 186 min: T_opt ≈ √(1,860) ≈ 43 min
C = 1 min, M = 30 min (100k GPUs): T_opt ≈ √60 ≈ 8 min
C = 0.1 min (async), M = 30 min: T_opt ≈ √6 ≈ 2.4 min
Read the rows in pairs: cutting the checkpoint cost with asynchronous writes lets the interval shrink, and a shorter interval is what keeps the lost fraction small as the fleet grows. At 100,000 GPUs a synchronous 5-minute checkpoint is untenable (T_opt ≈ 17 min, and each stop loses 8 to 9 minutes plus restart, against a 30-minute M, so the run spends over half its time not learning); an asynchronous checkpoint every couple of minutes keeps it above 80%. That single derivation is why checkpoint I/O design and fault-tolerant training get their own pages.
What the restart cost is made of
R is the term platform teams can attack after checkpointing, and it is a sum: detection (a hung collective can take the full NCCL timeout, ten minutes by default, to be noticed, which is why straggler and hang detection is its own discipline), drain and reschedule (seconds with a hot spare, minutes if a node must be found), checkpoint load (a 405B model's state is several terabytes across the ranks; from a parallel filesystem at hundreds of GB/s aggregate that is tens of seconds, from object storage minutes), collective rebuild (NCCL init across thousands of ranks, a minute or two), and warm-up. Llama 3's three human-handled incidents against 416 automatic ones show the goal: R is a property of the automation, not of the on-call.
Working it in the room
"How often will a 10,000-GPU run stop, and what does that mean for checkpointing?" wants the per-GPU rate from the public data (about 2 × 10⁻⁵ per GPU-hour), the fleet interval (50,600 ÷ 10,000 ≈ 5 h), the loss per stop (T/2 + R), and the Young-Daly interval for the checkpoint cost you can achieve; then the statement that above 90% effective time needs asynchronous checkpoints and automated restart. The follow-up held back is "your fleet is a different generation; how does the number change?", answered by saying the per-device rate is what to measure and the derivation is what transfers. The answer that sounds right and fails is quoting "MTBF of years per GPU" as reassurance without dividing by the fleet.
What to remember
- Llama 3: 16,384 H100s, 54 days, 419 unexpected stops (78% hardware, GPUs and HBM first), 3 needing a human, over 90% effective training time.
- Fleet interval M ≈ 50,600 GPU-hours ÷ N: 3.1 h at 16k GPUs, 30 min at 100k.
- Loss per stop ≈ T/2 + R; effective time ≈ 1 − (T/2 + R)/M − C/T.
- Young-Daly: T_opt ≈ √(2CM). Cheaper checkpoints allow shorter intervals, which is what keeps big fleets productive.
- R is detection plus reschedule plus checkpoint load plus collective rebuild; it is a property of the automation.