Trace one request through an inference platform. What are the spans, what does each cost, and where does latency actually hide?

Seven spans from arrival to the last token, with realistic durations for each. The three places latency hides that a naive trace misses entirely, why the decode phase is one span and not four hundred, and the attribute that explains most tail latency in a single field.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🔌 Networking & Storage

Debugging a Slow All-ReduceA training job reports its all-reduce at a third of what the fabric should deliver, every node passed its health check, and nothing is logged. This page is the isolation order that finds the cause in an hour instead of a day: measure the collective in isolation, split the job until the slow pair or rank appears, then check the specific things that make a link, a node or a placement slow. Most cases end at one NIC, one topology mismatch, or GPUDirect silently off.

Foundational

🩺 Fleet Reliability & Observability

GPU Failure Modes and XID ErrorsWhen a GPU misbehaves, the NVIDIA driver writes an XID line to the kernel log, and the number on that line is the first and often the only clue to what happened. Fleet engineers learn a dozen of them the way doctors learn a dozen lab values: 13 and 31 are almost always the application, 48 and 95 are memory that needs a reset, 63 and 64 are the row remapper reporting or failing, 74 is the NVLink fabric, 79 is a GPU that has vanished from the PCIe bus. This page gives the taxonomy, the decision for each (retry, reset, drain, RMA), and the derivation of how often a big fleet should expect each.

Core

🩺 Fleet Reliability & ObservabilitySign in

DCGM and GPU TelemetryNVIDIA's Data Center GPU Manager reads a GPU's counters, runs its diagnostics and exports both to the monitoring stack, and nearly every fleet's dashboards and alerts are built on it. The skill is knowing which of its hundreds of fields carry signal: the profiling metrics that say whether the tensor cores are busy (not the utilization number everyone reads first), the error counters that predict a failure, the throttle reasons that explain a slow step, and the diagnostic levels that decide whether a node returns to the pool. This page walks those fields, derives an MFU estimate from them, and gives a fleet's alert thresholds.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

ECC, Row Remapping and Memory ErrorsHBM stacks flip bits, and the difference between a fleet that shrugs and one that loses a training step to corruption is error-correcting codes plus the machinery that retires bad memory before it produces a double-bit error. A single-bit error is corrected silently and counted; a double-bit error is detected, kills the process, and on Ampere and later triggers the row remapper to swap the failing row for a spare at the next reset. This page explains the codes, the remapper's states, how to read the counters as a prediction of failure, and the RMA rules a fleet applies.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackOne serving replica has a per-token latency 40 percent worse than its peers. Find out why.Next in this trackWrite the policy for when a GPU is replaced rather than returned to service. What are the triggers and what do they cost?Next in this trackHow much spare capacity does a 16,000-GPU fleet need, and what are you actually reserving it for?

FEDITOR'S NOTE

Scored on the span structure with realistic durations, on queue wait and cache misses as the hidden costs, and on the decode phase as one span with per-token attributes rather than a span per token.

DISCUSSION · 0

No comments yet — be the first to share your approach.