Name the four collectives a training job uses, say what each moves, and match them to the parallelism that needs them.

TL;DR: All-reduce combines a value across every rank and gives every rank the result, which is what data parallelism does with gradients. Reduce-scatter combines and gives each rank one slice of the result, and all-gather is its mirror, collecting slices so everyone holds the whole. Those two compose into an all-reduce, which is exactly how ring implementations work and why sharded optimizers cost the same traffic as plain data parallelism. All-to-all has every rank send a different piece to every other rank, which is what expert parallelism needs to route tokens and what sequence parallelism needs to redistribute along the sequence dimension. For a message of S bytes across N ranks, each rank moves about S(N-1)/N for reduce-scatter, the same for all-gather, twice that for all-reduce, and about S(N-1)/N for all-to-all. For a 70B model's gradients at 140 GB, the all-reduce is about 280 GB per rank, which is why it is bucketed and overlapped rather than run once at the end.

How to approach it

Define each collective by what it does to the data rather than by its name, then give the per-rank bytes, then attach each to the parallelism that generates it. Do one worked example at a real model size so the numbers stop being symbolic. Close with the decomposition, because it explains a result that surprises people about sharded optimizers.

A strong answer

A typical situation: a team enables a sharded optimizer expecting communication to fall along with memory, and measures the same time in collectives as before. Memory per rank did drop by a large factor. The bytes on the wire did not, because an all-reduce was replaced by a reduce-scatter plus an all-gather, which is the same traffic split into two operations.

The four, with what each does and what it costs:

The worked example, so the table is not abstract:

model      70B parameters, gradients in bf16 = 140 GB
ranks      N = 1,024, per-rank network bandwidth B = 50 GB/s (a 400 Gb/s NIC)

plain data parallelism, one all-reduce per step:
  per-rank bytes = 2 x 140 GB x 1,023 / 1,024 = 279.7 GB
  time           = 279.7 / 50 = 5.6 s
  a training step at this scale is a couple of seconds, so an unoverlapped reduction would
  more than triple it

sharded (ZeRO or FSDP), reduce-scatter then all-gather:
  reduce-scatter = 140 x 1,023 / 1,024 = 139.9 GB
  all-gather     = 139.9 GB
  total          = 279.7 GB, identical to the all-reduce
  what changed is memory: each rank stores 1/1,024 of the optimizer state, not the traffic
sanity: this is the scenario above. Sharding is a memory technique, and anyone expecting it
        to cut communication has confused the two axes

ZeRO and FSDP covers the memory side, and Communication Volume Estimates has the general formulas. The reason the two forms cost the same is the decomposition: a ring all-reduce is literally implemented as a reduce-scatter followed by an all-gather, so the sharded version is the same wire traffic with the intermediate exposed.

Which parallelism produces which, in the order a large run uses them:

data parallel        all-reduce of gradients once per step (bucketed, overlapped with backward)
                     or reduce-scatter plus all-gather when the optimizer is sharded
tensor parallel      all-reduce of activations at two points per transformer layer, inside the
                     node on NVLink, on the critical path of every forward and backward
pipeline parallel    point-to-point sends of activations between adjacent stages, not a
                     collective at all, which is why it tolerates slower links
expert parallel      all-to-all to send each token to the GPU holding its expert, and a second
                     all-to-all to bring the results back, twice per MoE layer
sequence parallel    all-to-all to switch between splitting along sequence and splitting along
                     heads, or all-gather depending on the scheme

Tensor Parallelism and Expert Parallelism for MoE cover why those two are the demanding ones: both sit on the critical path of every layer rather than once per step, which is the reason tensor parallelism stays inside an NVLink domain and data parallelism can cross the slower fabric.

The frequency is as important as the volume, and it is what candidates most often omit:

per step, 70B model, 80 layers, TP 8, DP 128:
  data-parallel all-reduce      1 per step,  279.7 GB per rank, overlappable with backward
  tensor-parallel all-reduce    2 per layer x 80 layers = 160 per step, each small but on the
                                critical path with nothing to overlap it with
  a TP all-reduce of a 4,096-token activation at hidden 8,192 in bf16 is 67 MB, and at
  900 GB/s over NVLink takes about 150 us; 160 of them is 24 ms per step
sanity: the data-parallel reduction moves 4,000 times more bytes and hurts less, because it
        overlaps and the tensor-parallel one cannot

Measuring each of these is one tool with one subtlety. The nccl-tests binaries provide a benchmark per collective (all_reduce_perf, all_gather_perf, reduce_scatter_perf, alltoall_perf), and each prints two bandwidth columns. Algorithm bandwidth is message size divided by time, which is not comparable across collectives because they move different amounts for the same message. Bus bandwidth already applies the factor in the table above, so it is the number to compare against the link's rated speed and against other collectives. A bus bandwidth well below line rate on all-reduce but fine on all-gather points at the reduction path rather than at the fabric.

The reversal condition: these byte counts assume a ring or a ring-equivalent implementation, which is the right model for a large message on a standard fabric. On a system with in-network reduction, where a switch combines the data as it passes, the all-reduce cost falls toward a single S/B rather than 2S(N-1)/N, and the arithmetic above overstates it. Check whether the fabric does that before quoting the ring number in a capacity plan.

What interviewers probe next

• "Why is all-reduce twice the cost of all-gather?" It is a reduce-scatter plus an all-gather, each of which moves S(N-1)/N per rank.
• "Why does pipeline parallelism not appear in the table?" Its communication is point-to-point activation passing between adjacent stages, so it needs bandwidth but no collective and tolerates a slower link than the other two.
• "What makes all-to-all harder on a fabric than all-reduce?" Every rank talks to every other rank at once rather than to neighbors in a ring, so it stresses bisection bandwidth and produces incast at the receivers.
• "Where does the (N-1)/N factor come from?" A rank does not need to send its own slice to itself, so it exchanges N-1 of the N pieces.

Common mistakes

• Expecting a sharded optimizer to reduce network traffic, when it reduces memory and leaves the bytes unchanged.
• Quoting message volume without frequency, which hides that a small tensor-parallel collective 160 times a step can cost more than one large reduction.
• Calling pipeline parallelism's activation passing a collective.
• Using the all-reduce formula on a fabric that reduces in the network, where it overstates the cost.

Key takeaways

• Per rank, for S bytes over N ranks: reduce-scatter and all-gather each move S(N-1)/N, all-reduce moves twice that, all-to-all about S(N-1)/N.
• All-reduce decomposes into reduce-scatter plus all-gather, which is why sharded optimizers save memory and not bandwidth.
• 70B gradients at 140 GB over 1,024 ranks is 280 GB per rank, about 5.6 s at 50 GB/s, so it must overlap the backward pass.
• Frequency matters as much as volume: 160 small tensor-parallel all-reduces per step can cost more than one large data-parallel one, because only the second overlaps.