TL;DR: An XID is a numbered error the NVIDIA driver writes to the kernel log when a GPU reports a fault. The number matters because it separates three cases with completely different responses. Application faults, principally 13, 31 and 43, mean a kernel did something illegal such as an out-of-bounds access: the job failed, the hardware is fine, and the action is to tell the user. Recoverable hardware events, notably 63 and 94, mean memory errors were contained and a row is pending remapping: the action is to schedule a drain and a reset, not to panic. Fatal hardware faults, principally 48, 79 and 95, mean an uncorrectable error, a device that has fallen off the bus, or an uncontained memory error: the job is dead, the node must be drained immediately, and repeated occurrences justify a hardware return. Everything after that is automation: a log watcher matching the code, a policy table mapping code to action, and a controller that drains without waking anyone.
How to approach it
Explain what the code is in one sentence, then group by the action required rather than by number, because a list of thirty codes is not usable and three groups are. Give the handful worth memorizing in each group with what they mean physically. Then the automation, since the point of knowing the codes is that a machine acts on them.
A strong answer
A typical situation: a shared cluster's users complain about random job failures. The logs are full of XID 13 and XID 31, and the platform team spends a week investigating hardware. Both codes indicate illegal memory access by the running kernel, so the hardware was never at fault and the answer was in the code number all along.
The three groups:
group 1: application faults. The hardware is healthy; a kernel misbehaved.
13 Graphics Engine Exception, commonly an out-of-bounds or misaligned access
31 GPU memory page fault: an illegal address, the most common one in practice
43 GPU stopped processing, usually following an application error
action: fail the job, report to the user with the code, do not touch the node
the tell: they follow a user's job start and stop when that job stops
group 2: recoverable hardware events. Real, but not immediately fatal.
63 ECC page retirement or row remapping recorded: a memory region was retired
64 the remapping failed, which escalates this to group 3
92 high single-bit error rate: correctable errors are frequent, a leading indicator
94 contained ECC error: an uncorrectable error confined to one process
action: schedule a drain at the next checkpoint, reset the GPU, verify the remap, return
to service if the health suite passes; count occurrences per device over time
group 3: fatal hardware faults. The node is out until it is fixed.
48 double-bit ECC error: uncorrectable, the data was wrong
79 GPU has fallen off the bus: the device is not responding on PCIe at all
95 uncontained ECC error: the error escaped the process boundary
74 NVLink error: a link fault, sometimes recoverable but treated as fatal when repeated
119, 120 GSP RPC timeout: the GPU's management processor stopped responding
action: drain immediately, do not attempt to reuse, reboot the node, run the long
diagnostic, and return the device if the fault recurs
GPU Failure Modes and XID Errors has the full taxonomy; ECC, Row Remapping and Memory Errors explains what remapping is and why group 2 becomes group 3 when it fails.
Reading them:
where they appear the kernel log: dmesg, /var/log/kern.log, or the systemd journal
also exposed as a DCGM field, which is how a fleet collects them without
parsing text on every node
the line NVRM: Xid (PCI:0000:1a:00): 48, pid=12345, ...
the PCI address identifies the device, so a per-GPU count is possible
what to record code, device, timestamp, and the job that was running, since group 1 needs
the job and group 3 needs the device history
The automation, which is the actual answer to a platform question:
the policy table is the artifact: a mapping from code to action, reviewed by the team,
version-controlled, and applied by a controller rather than by a person
what it prevents: the two failure modes of manual handling, which are a group 3 code that
sits unnoticed while jobs keep landing on a bad device, and a group 1 code that triggers a
node drain because somebody pattern-matched "XID" to "hardware problem"
what to alert on: a group 3 code (page), a group 2 code count crossing a threshold on one
device (ticket), and a rate of group 1 codes from one user (tell the user, not the on-call)
sanity: at 2e-5 failures per GPU-hour, a 16,384-GPU fleet produces roughly 8 group 2 and 3
events a day. A policy that pages on all of them is 8 pages a day; one that pages only on
group 3 is closer to 2, which is the difference between a rota that works and one that burns out
Node Health Checks and Burn-In is the suite in the diagram; Incident Response for GPU Fleets is where the paging policy lives.
The reversal condition: the group assignments hold for the common codes, and a few are ambiguous enough that a fleet should record its own experience rather than trust a table. XID 74 is the clearest example: a single NVLink error after a topology change may be benign, while a recurring one on the same link is a failing connector. The general rule is that the code tells you what happened and the device's history tells you what to do about it, so a controller that acts on a single occurrence without consulting the device's counter will both over-drain and under-drain. Keep the per-device history and make the policy a function of both.
What interviewers probe next
- "How do you tell a user's bug from a hardware fault when both are possible?" Run the same job on a different node. A fault that follows the job is the code group 1 says it is, and a fault that stays with the device is hardware.
- "What is the difference between 94 and 95?" Contained versus uncontained: 94 means the uncorrectable error affected only one process and the rest of the GPU can continue, 95 means it escaped and the device state is untrustworthy.
- "Why does 79 require a reboot?" The device has stopped responding on PCIe, so the driver cannot reset it in software; the host must re-enumerate the bus.
- "Should you page on every XID?" No. Group 1 goes to the user, group 2 becomes a ticket, and only group 3 pages, which is what keeps the rota survivable at fleet scale.
Common mistakes
- Treating every XID as a hardware fault, which drains healthy nodes for users' kernel bugs.
- Ignoring group 2 codes because nothing broke, then hitting a double-bit error on a device that had been warning for a week.
- Acting on a single occurrence without the device's history, particularly for NVLink errors.
- Parsing kernel logs per node by hand instead of collecting the codes as a fleet-wide metric.
Key takeaways
- Three groups by action: application faults (13, 31, 43), recoverable hardware (63, 92, 94), fatal hardware (48, 79, 95, 74, 119).
- Group 1 fails the job and leaves the node in service; group 2 schedules a drain and reset; group 3 cordons immediately.
- The code says what happened; the per-device history says what to do, so keep a counter per device.
- Page only on group 3, ticket group 2, and report group 1 to the user who caused it.