← 🩺 Fleet Reliability & Observability
Advanced
ECC, Row Remapping and Memory Errors
HBM stacks flip bits, and the difference between a fleet that shrugs and one that loses a training step to corruption is error-correcting codes plus the machinery that retires bad memory before it produces a double-bit error. A single-bit error is corrected silently and counted; a double-bit error is detected, kills the process, and on Ampere and later triggers the row remapper to swap the failing row for a spare at the next reset. This page explains the codes, the remapper's states, how to read the counters as a prediction of failure, and the RMA rules a fleet applies.
Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew
RELATED CONCEPTS
LESSONS THAT TEACH THIS
PRACTICE THIS IN REAL QUESTIONS
GPU Fleet Reliability & ObservabilityOne GPU's correctable memory error rate has been climbing for a week. What does that predict and what do you do about it?→GPU & Accelerator ArchitectureWhat does ECC on a GPU cost you, and why do you keep it on across a fleet?→GPU Fleet Reliability & ObservabilityWrite the policy for when a GPU is replaced rather than returned to service. What are the triggers and what do they cost?→GPU Fleet Reliability & ObservabilityWhat is an XID error, which ones mean the hardware is bad, and which ones mean somebody's kernel has a bug?→GPU Fleet Reliability & ObservabilityWith DCGM available on every node, what do you actually collect, what do you alert on, and what do you deliberately ignore?→GPU Fleet Reliability & ObservabilityHow do GPUs actually fail at fleet scale, how often, and which failures should the platform expect to handle every day?→
