How do GPUs actually fail at fleet scale, how often, and which failures should the platform expect to handle every day?

TL;DR: The Llama 3 training run reported 419 unexpected interruptions over 54 days on 16,384 H100s, with roughly 78% attributed to hardware. That works out to about 0.32 interruptions per hour for the fleet, or roughly 2 times 10 to the minus 5 per GPU-hour, and a mean time between interruptions of about 3 hours. The component breakdown puts GPU faults including NVLink at about 30% and high-bandwidth memory at about 17%, with the remainder spread across host, network, storage and software. Two conclusions follow. First, at 16,384 GPUs something fails roughly every three hours, so a training run must checkpoint and resume automatically rather than page a human. Second, the same rate at 512 GPUs is one interruption every four days, which is why practices that look excessive at small scale become mandatory at large scale, and why teams that grow a cluster tenfold are surprised by an operational burden that grew tenfold with it.

How to approach it

Start from published numbers rather than intuition, since this is a question where a real dataset exists and quoting it is the difference between an answer and a guess. Derive the per-GPU-hour rate so it can be applied to any fleet size. Give the component breakdown. Then draw the two operational conclusions, because the numbers exist to change what the platform does.

A strong answer

A typical situation: a team runs 512 GPUs successfully for a year with manual restarts, expands to 8,192, and finds that the same practice now consumes an engineer's full attention. Nothing got less reliable per GPU. The fleet-wide rate scaled with the fleet, and a process that handled one failure every few days cannot handle one every few hours.

The rate, derived from the published record:

source     the Llama 3 405B pre-training run, as reported in its paper
fleet      16,384 H100 GPUs
duration   54 days of training
events     419 unexpected interruptions (excluding planned maintenance)

fleet rate           = 419 / (54 x 24 h) = 419 / 1,296 h = 0.323 interruptions per hour
mean time between    = 1 / 0.323 = 3.1 hours
per GPU-hour         = 0.323 / 16,384 = 1.97e-5, call it 2e-5
per GPU-year         = 2e-5 x 8,760 = 0.17, so about one interruption per GPU every 6 years

applying it to other fleet sizes:
  512 GPUs    2e-5 x 512   = 0.010 per hour -> one every 98 hours, about 4 days
  4,096 GPUs  2e-5 x 4,096 = 0.082 per hour -> one every 12 hours
  16,384      0.32 per hour                 -> one every 3 hours
  100,000     2.0 per hour                  -> one every 30 minutes
sanity: the per-GPU number is small and unremarkable; it is multiplication by fleet size that
        turns it into an operational regime change. That multiplication is the whole insight

Training Uptime and Interruption Statistics has the source data and the goodput consequences.

What breaks, by share of interruptions:

what this ordering means for where to spend effort:
  memory and GPU faults together are roughly half, and both are detectable before they become
  fatal: single-bit error rates rise before a double-bit error, NVLink replay counts rise
  before a link fails. So a health system that watches trends catches a large share early
  the remainder are mostly not predictable, which is what checkpointing is for
sanity: about half predictable and half not is the ratio that justifies building both a
        health-monitoring path and an automatic-restart path, rather than choosing one

GPU Failure Modes and XID Errors covers the specific fault taxonomy; ECC, Row Remapping and Memory Errors covers the memory path that is the largest single predictable category.

The two operational conclusions:

1. failure handling must be automatic
   at one interruption every 3 hours, a process requiring a human decision consumes the
   on-call rota and adds the human's response time to every event
   what automatic means: detect, drain the node, restart the job from its last checkpoint on
   replacement capacity, and file a ticket. The human sees the ticket, not the outage
   the cost of not doing this: at 20 minutes of lost work per event and 8 events a day, about
   2.7 hours a day of a 16,384-GPU fleet, which is roughly $27,000 a day at $2.5 per GPU-hour

2. spares are a capacity-planning input, not a contingency
   failures per day at 16,384 GPUs: 0.323 x 24 = 7.8 events, of which some fraction require
   physical replacement rather than a reboot
   with a 3-day repair turnaround, the steady-state number of GPUs out of service is
   failures per day x repair days x GPUs per event, which for whole-node replacement is
   a few nodes continuously
   so a fleet needs 2 to 3% spare capacity permanently, budgeted rather than borrowed

The reversal condition: these rates come from one large, well-instrumented run on one hardware generation, and they are the best public anchor rather than a universal constant. A fleet's own rate can differ by a factor of two in either direction depending on hardware batch, cooling, power quality and how aggressively the health system drains marginal nodes. Use the published figure to size a plan before you have data, then replace it with your own measured rate as soon as you have a quarter of history, and treat a persistent factor-of-two difference from the published number as something to investigate rather than accept.

What interviewers probe next

• "Does the rate change over a GPU's life?" Yes, in the usual pattern: elevated early from manufacturing defects, low through the middle, rising with age. Burn-in exists to move the early failures before production.
• "What fraction need physical replacement?" A minority. Many interruptions are recoverable by reset or reboot; replacement is driven mostly by memory errors that exhaust remapping and by devices that fall off the bus repeatedly.
• "How does this compare to CPU fleets?" Far higher per device, because these are large dies running near their thermal and power limits with high-bandwidth memory stacked on them.
• "What is the difference between an interruption and a failure?" An interruption stops the job; a failure damages hardware. Most interruptions are not failures, which is why automatic restart recovers most of the loss.

Common mistakes

• Quoting a mean time between failures for one GPU and forgetting to multiply by fleet size.
• Treating each interruption as an incident with a human response, which does not survive past a few thousand GPUs.
• Planning spares as a contingency rather than as a permanent 2 to 3% of capacity.
• Assuming the published rate applies exactly to your fleet without measuring your own.

Key takeaways

• Published anchor: 419 interruptions in 54 days on 16,384 GPUs, about 78% hardware.
• That is 0.32 per hour for the fleet, roughly 2 times 10 to the minus 5 per GPU-hour, and about 3 hours between interruptions.
• GPU faults including NVLink are about 30% and high-bandwidth memory about 17%, so roughly half are predictable from trends.
• At that rate handling must be automatic, and 2 to 3% spare capacity is a permanent budget line.