What do NVLink errors look like in telemetry, when is a link degrading rather than broken, and when do you drain the node?

A link that corrupts occasionally does not fail; it retransmits, and each retransmission costs bandwidth that shows up as a slow job with no error anywhere. The counters that reveal it, the arithmetic turning a replay rate into lost bandwidth, and the threshold where watching becomes draining.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

NVLink and Fabric FaultsThe links between GPUs are the part of a training node with the most connectors, the highest signalling rates and the least forgiveness: one marginal NVLink cable or one NVSwitch port turns an eight-GPU node into a straggler that slows a thousand-GPU job, and the symptom arrives as an NCCL timeout three layers away from the cause. This page covers what the links are, what their error counters mean, how a fault shows up in NCCL and in step time, how to isolate it to a GPU, a cable or a switch, and the arithmetic of why one degraded link is a whole-job problem.

Advanced

🕸️ Distributed Training🔒 Premium

Tensor ParallelismTensor parallelism splits individual weight matrices across GPUs so each rank computes a slice of every layer, which is how a model whose single layer does not fit one GPU gets trained at all. It costs four all-reduces per transformer block on the critical path, which is why it stays inside the NVLink domain and rarely exceeds 8 ranks.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

NVLink, NVSwitch and PCIeInside a node, GPUs talk over NVLink at 900 GB/s per H100 through an NVSwitch fabric that gives all eight cards full bandwidth to each other; to the host and to anything outside the node they talk over PCIe at 64 GB/s or a 400 Gb/s NIC at 50 GB/s. That fifteen-fold gap is why tensor parallelism stays inside the eight-GPU domain, why NVL72 changes the serving math for MoE, and why the question "how many GPUs share an NVLink domain?" is the first thing to ask about any cluster.

Foundational

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL is the library every PyTorch collective lands in, and its choice of ring or tree, channel count and protocol decides whether an all-reduce runs at fabric speed or at a third of it. Knowing what NCCL_DEBUG=INFO prints, and which environment variable changes which decision, is the difference between tuning a cluster and guessing at it.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWrite the postmortem for a training run that lost twelve hours. What goes in it, and what makes the action items stick?Next in this trackYou own the on-call rota for a GPU fleet. What is allowed to wake someone at 3am, and what must not?Next in this trackDesign the controller that decides whether a GPU node is healthy. What are its states, and how do you tune its false-positive rate?

FEDITOR'S NOTE

Scored on knowing replays degrade bandwidth silently before a link fails, on converting a replay rate into a throughput loss, and on a threshold policy that distinguishes a degrading link from a failed one.

DISCUSSION · 0

No comments yet — be the first to share your approach.