GPU Failure Modes and XID Errors
When a GPU misbehaves, the NVIDIA driver writes an XID line to the kernel log, and the number on that line is the first and often the only clue to what happened. Fleet engineers learn a dozen of them the way doctors learn a dozen lab values: 13 and 31 are almost always the application, 48 and 95 are memory that needs a reset, 63 and 64 are the row remapper reporting or failing, 74 is the NVLink fabric, 79 is a GPU that has vanished from the PCIe bus. This page gives the taxonomy, the decision for each (retry, reset, drain, RMA), and the derivation of how often a big fleet should expect each.
TL;DR: An XID is the driver's error code, one line in
dmesgof the formNVRM: Xid (PCI:0000:xx:00): NN, .... Read it in three groups. Application faults (13, 31, 43, 45): a kernel touched a bad address or timed out; the job dies, the GPU is fine, retry after checking the code. Memory faults (48, 63, 64, 92, 94, 95): ECC events; 94 is contained and the job restarts, 48 and 95 need a GPU reset, 63 needs a reset to apply a row remap, 64 means the remap failed and the board goes back to the vendor. Hardware and fabric faults (74, 79, 119, 120): NVLink errors, a GPU that fell off the bus, or the GSP firmware hung; drain the node, reset or reboot, and if it recurs, RMA. On a 16,000-GPU fleet, expect an unexpected hardware interruption roughly every three hours, most of them GPUs and their memory.
What an XID is
The NVIDIA kernel driver (nvidia.ko, logged as NVRM) reports errors it detects in the GPU as XID events: an integer code, the PCI address of the GPU, and some context. They appear in the kernel ring buffer and in /var/log/syslog or journalctl -k, and DCGM reports them as a field. The code tells you which subsystem detected the problem; the context (a channel id, an address, a link number) narrows it down; and the count and recurrence tell you whether it is an application bug, a transient, or a board that is dying.
[Tue Sep 1 03:12:44 2026] NVRM: Xid (PCI:0000:4e:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
[Tue Sep 1 03:12:44 2026] NVRM: GPU 0000:4e:00.0: GPU has fallen off the bus.
That line is the canonical fleet-page: the GPU at PCI 4e stopped responding to the host. nvidia-smi will hang or report the GPU missing, and every process using it is stuck.
The taxonomy
| XID | What it means | Who is at fault | What to do |
|---|---|---|---|
| 13 | graphics engine exception: a kernel faulted (illegal instruction, out-of-range access) | almost always the application | job dies; fix the kernel; retry on the same GPU |
| 31 | GPU memory page fault: a kernel read or wrote an unmapped address | the application (a bad pointer, an out-of-bounds index) in the vast majority of cases | same as 13; if it recurs across unrelated jobs on one GPU, suspect the board |
| 43 | GPU stopped processing: a channel was reset after an error | application, following a 13 or 31 | retry |
| 45 | preemptive cleanup after a prior error | a consequence, not a cause | look at the XID before it |
| 48 | double-bit ECC error (DBE): uncorrectable memory corruption detected | hardware (HBM) | GPU reset required; the driver also records a row remap; if DBEs recur, RMA |
| 63 | row remapping event recorded | hardware, handled | the remap applies at the next reset; schedule a reset; the GPU is fine afterwards |
| 64 | row remapping failure: a remap entry could not be recorded | hardware, and not automatically terminal | reboot first, since the remap may apply cleanly on the next reset; if it recurs, drain and run field diagnostics, then RMA on the vendor criteria |
| 74 | NVLink error: a link failed training or reported fatal errors | fabric (cable, connector, NVSwitch port, or the GPU's link block) | drain; reset; check the link counters; if it recurs, RMA the GPU or the switch tray |
| 79 | GPU has fallen off the bus: no response over PCIe | hardware, power or thermal | drain and reboot; if it recurs, RMA |
| 92 | high single-bit ECC error rate | hardware, early warning | watch; the remapper will handle it; RMA if the rate keeps climbing |
| 94 | contained ECC error (Ampere and later): the error was isolated to the affected context | hardware, contained | the job dies and restarts; the GPU keeps running; no reset needed |
| 95 | uncontained ECC error | hardware | GPU reset required; if it recurs, RMA |
| 119, 120 | GSP firmware RPC timeout or error (the GPU System Processor stopped responding) | firmware or hardware | reset; firmware update; if it recurs, RMA |
The first thing a good engineer does with an unknown code is look it up in NVIDIA's XID catalogue; the second thing is to check which of the three groups it falls in, because the group decides the action long before the exact code does.
The three decisions
Sorted by what you actually do, rather than by which subsystem reported it, the whole taxonomy fits on one card:
Retry, reset, or replace. Application faults (13, 31, 43, 45) get a retry after the code is checked: the GPU did what it was told. The fleet signal to watch is one GPU throwing 31s under many different jobs, which flips the diagnosis from application to hardware.
Memory faults split by containment. A 94 is the good case introduced with Ampere: the error is confined to the process that touched the bad page, the process is killed, the GPU continues and the page is retired. A 48 or 95 requires a GPU reset (nvidia-smi -r, which needs the GPU idle, or a node reboot on a full NVSwitch system where per-GPU reset is often not practical). A 63 means the remapper recorded a repair that applies on the next reset, so the node is scheduled for one at the next idle window. A 64 means a remap entry could not be recorded, and the tempting reading is that the board is out of spare rows and finished. That is a conclusion, not an observation: the failure to record is what the driver reports, and the cause may be transient. The vendor guidance is a reset or reboot first, because the pending remap frequently applies cleanly; then, if the error returns, drain the node and run the field diagnostics, which is what actually establishes whether the spare rows are exhausted. RMA follows the vendor criteria and that evidence, not the first occurrence. Treating a single 64 as an automatic return sends back working boards and, worse, trains a fleet team to skip the diagnosis (ECC, Row Remapping and Memory Errors).
Hardware and fabric faults get a drain, a reset or reboot, and a recurrence rule: the same GPU or link failing twice in a window (a week is common) goes to RMA rather than back into the pool. A 79 is the sharpest: the GPU is gone from the bus, so nothing short of a reboot brings it back, and a GPU that falls off the bus twice is almost always a board or a power delivery problem. A 74 points at the NVLink fabric (NVLink and Fabric Faults); the counters say whether it was the GPU end, the cable, or the switch.
How often, and why that number matters
The public reference point is Meta's Llama 3 training report: 16,384 H100s over 54 days saw 466 job interruptions, 419 of them unexpected, and about 78% of the unexpected ones were confirmed or suspected hardware. Of those, GPUs and their HBM were the largest share (faulty GPU about 30%, HBM about 17%), with software bugs, network switches and cables, host maintenance, GPU SRAM, and the GPU system processor behind them.
unexpected interruptions: 419 in 54 days
rate = 419 ÷ 54 ≈ 7.8 per day ≈ one every 3.1 hours across the whole fleet
per-GPU MTBF, treating every interruption as one GPU-attributable event (an overestimate of GPU blame):
MTBF_gpu ≈ 16,384 GPUs × 3.1 h ≈ 50,800 GPU-hours ≈ 5.8 GPU-years between failures
so on a fleet of N GPUs, the expected time between unexpected stops is
T ≈ 50,800 h ÷ N
N = 1,024: ≈ 50 h (about two days); N = 16,384: ≈ 3.1 h; N = 100,000: ≈ 30 min
sanity: a per-GPU lifetime of several years sounds fine, and at 100k GPUs it becomes a stop every
half hour, which is why checkpoint cadence and automatic restart dominate at scale.
This is the derivation behind every fleet reliability design: failures that are rare per device are continuous per fleet, and the arithmetic gives the interval that checkpointing and restart have to beat (Training Uptime and Interruption Statistics).
Working it in the room
"You see XID 79 on a node at 3am; walk me through it" wants the recognition (GPU fell off the bus), the immediate action (drain the node, requeue the job from its last checkpoint, reboot), the diagnosis (power, thermal, board; check the BMC log and the PCIe link state), and the recurrence rule (twice, and it is an RMA). The follow-up held back is "how do you tell an application page fault from a bad GPU?", answered with the pattern across jobs: one job faulting on many GPUs is code; many jobs faulting on one GPU is hardware. The answer that sounds right and fails is "reboot everything and move on": without the recurrence rule the same board comes back into the pool and stops the next job.
What to remember
- An XID is the driver's error code in
dmesg; read the group first: application (13, 31, 43, 45), memory (48, 63, 64, 92, 94, 95), hardware and fabric (74, 79, 119, 120). - 94 is contained (restart the job); 48 and 95 need a GPU reset; 63 applies a remap at the next reset; 64 is an RMA.
- 79 is a GPU gone from the bus: drain and reboot; twice and it is replaced. 74 is the NVLink fabric.
- One job faulting on many GPUs is code; many jobs faulting on one GPU is hardware.
- Llama 3's 419 unexpected interruptions in 54 days on 16,384 H100s give one every 3.1 hours and about 50,800 GPU-hours per failure; scale by N for your fleet.