Two identical 8-GPU jobs get 8 GPUs each. One runs at half the speed of the other. What did the scheduler do, and how do you stop it?

TL;DR: The slow job was placed as 4 GPUs on one node and 4 on another, so its tensor-parallel all-reduces, which happen twice per layer per micro-batch, leave the 900 GB/s NVLink domain and cross a 50 GB/s NIC; at that ratio the communication that was hidden under compute becomes the step time. The fix is a scheduler that knows the hierarchy (NVLink domain, rail leaf, spine block) and allocates whole nodes for anything with tensor parallelism, then searches outward for the rest, and waits rather than placing a job across a level it cannot afford.

How to approach it

Ask how the job is parallelized (TP8 on one node is the usual guess for 8 GPUs) and what fabric the nodes have. Then compute the per-layer all-reduce volume and divide it by the two bandwidths so the gap is a number. State the rule: TP inside the NVLink domain, DP across a rail, PP anywhere. Then the scheduler policy that enforces it, and the cost of that policy (fragmentation, packing) so the answer is a trade rather than a wish.

A strong answer

A typical situation: a researcher files a ticket that "node B is broken" because the same fine-tuning script runs at 2,100 tokens per second per GPU on one allocation and 1,000 on another. Every link on node B checks healthy. The difference is that the second allocation was not one node.

An 8-GPU job for a model that does not fit on one card is almost always tensor parallel across all 8. Tensor parallelism all-reduces the activations after every attention block and every MLP block, so twice per layer per micro-batch, and the volume is the activation tensor:

inputs:  70B-class model, hidden d = 8,192, 8k tokens per micro-batch, bf16 (2 B)
         layers L = 80; two all-reduces per layer
activation tensor per all-reduce = tokens × d × 2 B = 8,192 × 8,192 × 2 = 134 MB
ring all-reduce moves ≈ 2 × (N − 1) ÷ N × tensor ≈ 1.75 × 134 MB = 235 MB per GPU per all-reduce
per micro-batch: 2 × 80 × 235 MB = 37.6 GB per GPU

on NVLink (H100 SXM, 900 GB/s bidirectional per GPU; ~450 GB/s each way effective):
   37.6 GB ÷ 450 GB/s ≈ 0.08 s per micro-batch
split 4 + 4 across nodes, one 400 Gb/s NIC ≈ 50 GB/s in the path of every ring:
   the ring's slowest link is the NIC: 37.6 GB ÷ 50 GB/s ≈ 0.75 s per micro-batch
compute per micro-batch on 8 H100s at 40% MFU: 6 × 70e9 × 8,192 × 8 ÷ (8 × 989e12 × 0.40) ≈ 0.9 s
step time, single node:   max(0.9, 0.08) plus a little exposure ≈ 1.0 s
step time, split 4 + 4:   TP all-reduce is on the critical path: 0.9 + 0.75 ≈ 1.65 s, and in
                          practice worse because each layer waits on it synchronously
sanity: 0.75 ÷ 0.08 is a 9x slower collective, and it runs 160 times per micro-batch; the
        observed 2x step time is what "half hidden, half exposed" looks like on the dashboard

The per-parallelism rule falls out of the volumes. Tensor parallel and expert parallel (MoE all-to-all) must stay inside the NVLink domain. Data parallel and FSDP move a few hundred gigabytes per step but overlap with the backward pass, so one hop on a rail is fine and a spine block is tolerable. Pipeline parallel moves a hundred megabytes per micro-batch boundary and does not care. Topology-Aware Scheduling works the multi-node version; the 8-GPU case is the smallest instance of the same rule.

What the scheduler did: it treated 8 GPUs as an integer and found 4 free here and 4 free there, because the default scoring spreads pods across nodes for availability. That is right for web replicas and wrong for anything with tensor parallelism.

What stops it, in order of how much it changes:

1. Whole-node allocation for TP jobs: a pod requesting 8 GPUs fits only on a node with 8 free. On Kubernetes, make the job one pod per node (8 GPUs per pod) instead of 8 single-GPU pods, or use a pod affinity rule; on Slurm, --nodes=1 --gres=gpu:8.
2. A hierarchy the scheduler can see: node labels for rail leaf and spine block, or Slurm's topology.conf, and a scheduler that searches from the tightest level outward. Kueue's topology-aware scheduling expresses "required within host" or "preferred within rack" per workload.
3. Packing, so whole nodes exist: fill nodes before opening new ones, or the fleet ends up with 4 free GPUs on every node and no whole node anywhere.
4. Wait rather than degrade: a job that cannot get its shape holds the queue slot rather than running at half speed for twice as long, because the degraded run delays everyone behind it as well.

The cost of the policy is availability and fragmentation: a whole-node job loses everything when the node fails, and packing makes a node failure take out more of one job. For training that is the right trade, because a lost GPU restarts the job from checkpoint either way. The condition that reverses the whole-node rule is a job with no tensor parallelism at all (a 7B model fine-tuned with pure data parallelism), which can be spread across nodes at a small cost and should be if that is what keeps the queue moving.

The reversal condition: a job small enough to fit inside one node, where every placement is equivalent and topology awareness costs scheduling latency for nothing. Topology-Aware Scheduling is worth turning on only above that size, and nvidia-smi topo -m at job start confirms what you got. Communication Volume Estimates gives the bytes that make placement matter.

What interviewers probe next

• "How would you prove the placement is the cause?" nvidia-smi topo -m inside each allocation shows NV links versus a NIC path; NCCL debug logs at NCCL_DEBUG=INFO print the ring topology and which transport (NVLink, IB) each channel uses.
• "Does the rule change for an NVL72 rack?" The NVLink domain is 72 GPUs, so TP and EP can span the rack; the whole-node rule becomes a whole-rack-partition rule.
• "The cluster has no whole node free but eight nodes with 4 free each. Now what?" Wait, or run a defragmentation step; running the TP job split is the slow path the question started with.
• "What about 2 GPUs on one node and 6 on another?" Worse than 4 + 4 in a different way: the ring's slowest link still crosses the NIC, and the imbalance adds nothing; any split is the full penalty.

Common mistakes

• Blaming hardware ("node B is slow") when every link reads healthy; the topology differs, the hardware does not.
• Quoting NVLink and NIC bandwidths without multiplying by the collective count; 9x slower once is nothing, 9x slower 160 times per micro-batch is the step.
• Applying the whole-node rule to data-parallel jobs and then wondering why the queue is stuck.
• Assuming the scheduler knows topology; it knows only what the platform labels.

Key takeaways

• TP all-reduces run twice per layer; at 37.6 GB per GPU per micro-batch, NVLink takes 0.08 s and a NIC path 0.75 s against 0.9 s of compute.
• TP and EP stay in the NVLink domain; DP on a rail; PP anywhere.
• Enforce with whole-node allocation, a labeled hierarchy, packing, and waiting rather than degrading.
• Prove it with nvidia-smi topo -m and NCCL's transport logs, not with a node health check.