"Your All-Reduce Got Slow": How to Answer the Most Common Distributed Training Debug Question

Some version of this question shows up in nearly every distributed training interview: a training job that was running at about 40 percent model FLOPs utilisation is now at 22, the all-reduce looks slow, what do you do. The answer that scores is not a list of possible causes. It is a narrowing sequence where each step rules something out, and where the first step is arithmetic rather than a tool.

Step zero: work out what it should cost

Before you look at anything, establish the floor. If you cannot say what a healthy all-reduce would take, you cannot tell whether the current one is broken.

For a 70 billion parameter model in BF16, the gradient is about 140 GB. A ring all-reduce moves approximately 2 × (N-1)/N × the data volume per rank, so for large N call it 280 GB of traffic per rank across the operation. At 400 Gb/s per GPU, roughly 50 GB/s, the floor is about 5.6 seconds.

Now the number in front of you means something. If the collective is taking 6 seconds, it is at 93 percent of theoretical and there is nothing to debug on the network; your throughput problem is somewhere else entirely. If it is taking 20 seconds, you are at 28 percent of what the fabric can do and there is a real fault.

Saying this first changes the entire round. It tells the interviewer you know the difference between a bug and physics, and a meaningful share of real-world "slow all-reduce" reports turn out to be physics. Our napkin math track drills this arithmetic until it is automatic.

Step one: is it the collective, or is it a straggler?

This is the highest-value diagnostic in distributed training and most candidates skip it.

A collective is a synchronisation point. If one rank arrives late, every other rank sits inside the all-reduce waiting, and the time is attributed to the collective. From the job's average step time, a straggler and a genuine network problem look identical.

So get per-rank step times, or per-rank arrival times at the collective. If 1,023 ranks are fast and one is slow, you do not have a network problem, you have one sick GPU or one sick host, and the fix is to find and drain it.

Straggler causes worth naming: thermal throttling on a badly cooled node, a GPU accumulating ECC errors and retrying, a noisy neighbour saturating shared storage during the dataloader phase, a CPU affinity or NUMA misconfiguration on one host, or a rank whose data shard is genuinely larger.

Our reliability and observability questions cover the telemetry side, including what device-level metrics actually tell you about throttling and errors.

Step two: microbenchmark the fabric

If all ranks are slow together, separate the fabric from the job. Run the standard collective microbenchmark at the same message size and the same topology as your job.

The result splits the problem cleanly:

The microbenchmark hits near line rate but the job does not. The fabric is fine and the problem is in your job. Look at whether communication is overlapping with the backward pass at all, whether gradients are being bucketed into reasonably sized messages (many tiny all-reduces are latency-bound and will never reach bandwidth), whether there are CPU-side launch gaps, and whether something is forcing a synchronisation each step.

The microbenchmark is also slow. The fabric has a fault. Go to step three.

This step is worth naming explicitly in an interview because it is the move that turns an open-ended problem into a bisected one.

Step three: find the bad link

When the fabric itself is slow, the causes cluster into a short list.

A degraded link. A port that has negotiated down to a lower rate, or one accumulating symbol errors and retransmitting. Everything reports as up, nothing alerts, and one rail crawls. Because a ring passes through every rank, a single bad link sets the pace for the entire collective. Check link rates and error counters across the fabric, not just on the hosts you suspect.

A topology change. A rescheduled job landed on a worse placement. Ranks that used to communicate over NVLink inside a node now traverse the fabric, or the allocation spans more switch hops than it did. Nothing is broken; the placement is worse. This is why topology-aware and gang scheduling matter, and it is a good moment to say so.

Path collisions. Flows hashing onto the same path, leaving some links saturated while others idle. Rail-optimised designs exist to make this less likely, and adaptive routing exists to fix it when it happens anyway.

Congestion and flow control. Pause frames, congestion notifications, and a fabric that is technically healthy but oversubscribed at the spine given what else is running on it.

A software change. A driver update, a communication library version bump, or a changed environment variable that made the library pick a different algorithm or fall back to a slower transport. If this started overnight, something changed overnight; ask what.

Our networking and storage track works through fabric topology, congestion and the counters worth reading.

Step four: confirm what the library is actually doing

Rather than reciting flags, describe what you want to see: which algorithm and protocol the collective library chose, which transport each connection is using, and whether the ring or tree it built matches the physical topology you expect. A run with debug output enabled will print the topology it constructed and the transport per peer pair, and a peer pair that has silently fallen back from a direct high-speed transport to a slower path is a common and very findable cause.

That framing reads as someone who has done this. A memorised list of environment variables does not.

The narrowing sequence, in one place

1. Compute the floor. What should this collective cost? Is the observed number a bug or physics?
2. Per-rank step times. One slow rank means a straggler, not a network problem.
3. Microbenchmark. Fabric healthy means the problem is in the job; fabric slow means go hunting.
4. Link rates and error counters, placement, routing, congestion. Find the degraded link or the bad placement.
5. Check what the library chose, and check what changed recently.
6. Fix, then verify against the floor from step one, because "it feels faster" is not a result.

What the interviewer is scoring

Not whether you name the actual cause. They usually have one in mind, but the score comes from the shape of your search. Specifically: do you quantify before you investigate, do you separate the straggler case from the network case, do you bisect with a microbenchmark rather than guessing, and do you say what each observation would rule out.

One more thing worth doing at the end: say what you would change so this is detected automatically next time. Per-rank step time distributions, link rate and error counters as first-class fleet metrics, and an alert on achieved bus bandwidth against the expected floor. Turning an incident into a permanent default is the difference between a senior answer and a staff one, as the leveling signals page explains.

Practise the underlying material in the distributed training track, and use the must-know questions to find which step of the sequence you are weakest on.