You see 'Watchdog caught collective operation timeout' in the logs. What happened, and how do you find which rank caused it?

The watchdog says an all-reduce ran for ten minutes and never finished. That means one rank never entered it, entered a different one, or died. How to read the log line, the flight recorder that names the rank in seconds, the env vars that make the next one debuggable, and why the timeout is a symptom every time.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Stragglers and HangsSynchronous training runs at the speed of its slowest rank, so one GPU that is 30% slow makes a thousand GPUs 30% slow, and one rank that never arrives at a collective makes the other 1,023 wait in silence until a watchdog fires ten minutes later. Finding the slow rank and the stuck rank is the most common on-call task on a training fleet, and the tooling for it (per-rank timing, the NCCL flight recorder, stack dumps across ranks) is specific and learnable. This page derives the straggler tax from first principles, lists the causes in the order they actually occur, and gives the procedure for a hang.

Foundational

🔌 Networking & Storage

Debugging a Slow All-ReduceA training job reports its all-reduce at a third of what the fabric should deliver, every node passed its health check, and nothing is logged. This page is the isolation order that finds the cause in an hour instead of a day: measure the collective in isolation, split the job until the slow pair or rank appears, then check the specific things that make a link, a node or a placement slow. Most cases end at one NIC, one topology mismatch, or GPUDirect silently off.

Advanced

💻 Coding for Infra🔒 Premium

Batching Queues and BackpressureWrite a request batcher is the coding round's version of the serving engine's scheduler: requests arrive one at a time, the GPU wants them in groups, and the batcher decides when a group is full enough to send without holding anyone too long or accepting more than it can hold. The two knobs are the maximum batch size and the maximum wait, the invariant is a bounded queue, and the follow-ups (priorities, cost-aware batching, cancellation, bounded in-flight batches) are the ideas the real engines carry. This page implements the batcher in asyncio, derives what each knob buys, and walks the follow-ups.

Foundational

🕸️ Distributed Training

Collective Communication PrimitivesAll-reduce, all-gather, reduce-scatter, all-to-all and broadcast are the five operations every parallelism strategy is built from, and each has a fixed per-rank traffic cost you can compute before a job runs. Knowing those volumes for a named model is how you decide whether a layout is compute-bound or waiting on the network.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow do you overlap communication with compute in training, and where does the overlap stop working?Next in this trackHow is training on TPUs with JAX different from training on GPUs with PyTorch? What do you stop doing by hand?Next in this trackFine-tune a 70B on one 80 GB card. What do NF4 and double quantization actually buy, and where does DoRA change the arithmetic?

FEDITOR'S NOTE

Scored on treating the timeout as the effect of a mismatch or a dead rank rather than a network problem, on knowing the flight recorder and its env vars, and on finding the rank whose sequence number lags.

DISCUSSION · 0

No comments yet — be the first to share your approach.