How do you establish that the link between two GPU nodes is healthy, and what numbers should each test return?

Three tools measure three different things, and a link can pass one while failing another. The message-size sweep that separates a latency problem from a bandwidth problem, the number each test should return on a healthy 400 gigabit port, and the single test that catches the failure the others miss.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🖧 Hardware & Cluster Build-Out

Burn-In and Acceptance TestingNew hardware fails early or it fails late, and burn-in exists to move the early failures before the cluster is handed over rather than after. A proper acceptance test runs every layer under sustained load for days, compares every node against its siblings rather than against a specification, and produces a signed number the buyer and the vendor both agree on. The comparison is the important part: identical hardware running identical work should produce identical numbers, and the outliers are the finding.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Node Health Checks and Burn-InThe cheapest failure is the one caught before a job starts. A GPU fleet runs three kinds of check: a burn-in of days on new or repaired hardware to shake out the parts that die young, a pre-flight suite of minutes before a node is handed to a job, and continuous checks during the job. Each test has an expected number (a GEMM within a few percent of the fleet median, an all-reduce at rated bus bandwidth, a NIC at line rate, a memory test with zero errors), and a node that misses any of them goes to quarantine. This page gives the suites, the numbers, and the arithmetic that justifies the time they cost.

Foundational

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL is the library every PyTorch collective lands in, and its choice of ring or tree, channel count and protocol decides whether an all-reduce runs at fabric speed or at a third of it. Knowing what NCCL_DEBUG=INFO prints, and which environment variable changes which decision, is the difference between tuning a cluster and guessing at it.

Foundational

🔌 Networking & Storage

Debugging a Slow All-ReduceA training job reports its all-reduce at a third of what the fabric should deliver, every node passed its health check, and nothing is logged. This page is the isolation order that finds the cause in an hour instead of a day: measure the collective in isolation, split the job until the slow pair or rank appears, then check the specific things that make a link, a node or a placement slow. Most cases end at one NIC, one topology mismatch, or GPUDirect silently off.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhy do large training clusters provision roughly 400 gigabits per second per GPU rather than more or less?Next in this trackWhat is Ultra Ethernet trying to fix, and does it change how you would choose a fabric today?Next in this trackDesign the complete network for a 1,024-GPU cluster: compute fabric, storage, management. What are the counts and where does the money go?

FEDITOR'S NOTE

Scored on knowing what each tool measures and in what order to run them, on the expected values rather than vague healthiness, and on the message-size sweep as the thing that distinguishes the two failure classes.

DISCUSSION · 0

No comments yet — be the first to share your approach.