expert★ EssentialNewMetaAnthropicOpenAI

A multi-GPU training job hangs at step 400 with every GPU at 100 percent utilization. Debug it.

Full GPU utilization during a hang is the clue, because a spinning collective looks identical to real work. The isolation order that finds the mismatched rank in minutes, a runnable reproducer that hangs on demand, and the fix that gates the logging rather than the collective.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Stragglers and HangsSynchronous training runs at the speed of its slowest rank, so one GPU that is 30% slow makes a thousand GPUs 30% slow, and one rank that never arrives at a collective makes the other 1,023 wait in silence until a watchdog fires ten minutes later. Finding the slow rank and the stuck rank is the most common on-call task on a training fleet, and the tooling for it (per-rank timing, the NCCL flight recorder, stack dumps across ranks) is specific and learnable. This page derives the straggler tax from first principles, lists the causes in the order they actually occur, and gives the procedure for a hang.

Foundational

🕸️ Distributed Training

Collective Communication PrimitivesAll-reduce, all-gather, reduce-scatter, all-to-all and broadcast are the five operations every parallelism strategy is built from, and each has a fixed per-rank traffic cost you can compute before a job runs. Knowing those volumes for a named model is how you decide whether a layout is compute-bound or waiting on the network.

Foundational

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL is the library every PyTorch collective lands in, and its choice of ring or tree, channel count and protocol decides whether an all-reduce runs at fabric speed or at a third of it. Knowing what NCCL_DEBUG=INFO prints, and which environment variable changes which decision, is the difference between tuning a cluster and guessing at it.

Foundational

🔌 Networking & Storage

Debugging a Slow All-ReduceA training job reports its all-reduce at a third of what the fabric should deliver, every node passed its health check, and nothing is logged. This page is the isolation order that finds the cause in an hour instead of a day: measure the collective in isolation, split the job until the slow pair or rank appears, then check the specific things that make a link, a node or a placement slow. Most cases end at one NIC, one topology mismatch, or GPUDirect silently off.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Closest to this topicWhat is gang scheduling, and what goes wrong on a Kubernetes cluster that does not have it?Closest to this topicA training run hangs every few hours with no error, and the GPUs sit idle until the timeout fires. Find the cause.Closest to this topicYou see 'Watchdog caught collective operation timeout' in the logs. What happened, and how do you find which rank caused it?

FEDITOR'S NOTE

Scored on reading 100 percent utilization as a spinning kernel rather than progress, on the flight-recorder evidence of unequal collective counts per rank, and on the fix gating the print rather than the collective.

DISCUSSION · 0

No comments yet — be the first to share your approach.