TL;DR: Because collectives are barriers, the job runs at the pace of its slowest rank, so one node at half speed halves the whole job rather than costing a proportional fraction. Five causes account for nearly all cases, and each has a check that takes under a minute. Thermal or power throttling shows in the clock and throttle-reason telemetry. A degraded interconnect link shows in the link error and replay counters, with collective bandwidth low while compute is normal. Wrong processor affinity, where a process runs on the socket far from its GPU, shows in the process placement and produces a consistent penalty on host-to-device transfers. A noisy neighbor process shows in host load. And a degrading GPU shows in memory error counters and remapping state. Run the checks in that order because they take increasing effort, and use one decisive discriminator early: restart the job with that node excluded and see whether the slowness follows the node or stays with the job, which separates hardware from software in one step.
How to approach it
Establish first why one slow node matters disproportionately, since that is what justifies the urgency. Then give the checks in order of effort with what each rules out, because an unordered list is not a procedure. Introduce the follow-the-node test early, since it partitions the space in one move. Close with what to do when everything checks out clean, which is the case that separates people who have done this from people who have read about it.
A strong answer
A typical situation: a 128-node job's step time rises from 400 to 780 milliseconds overnight. Every node reports healthy, every GPU shows full utilization, and the aggregate metrics look normal because they are averages. Per-rank timing shows node 47 taking 780 milliseconds while everyone else takes 400 and then waits.
Why one node sets the pace:
128 nodes, healthy step 400 ms, one node at 780 ms
each collective is a barrier: no rank proceeds until all have arrived
job step time = max over ranks = 780 ms
throughput loss = 1 - 400/780 = 49%
the mean across ranks = (127 x 400 + 780) / 128 = 403 ms, a 0.7% change
sanity: the job lost half its speed and the average moved by less than a percent, which is
why this is invisible without per-rank timing and why it can persist for days
The checks, in order of effort, with what each eliminates:
1. clocks and throttle reasons (10 seconds)
read the current and maximum clocks and the throttle-reason bits for each GPU on the node
thermal or power throttling shows as clocks well below the rest of the fleet with a
corresponding reason bit set
rules out / in: thermal and power. A node throttling at half clock explains half speed
exactly, which makes this the cheapest and most often correct first check
2. interconnect link health (30 seconds)
NVLink error and replay counters on the node, and the fabric link counters for its ports
a degraded link shows as rising replays or a link running below its rated width
rules out / in: communication. Corroborate by comparing that node's collective time
against its compute time: if compute is normal and collectives are slow, it is the link
3. processor and memory affinity (30 seconds)
check that each rank's process is running on the socket nearest its GPU and that memory is
allocated locally, since a process on the far socket pays for every host-to-device transfer
rules out / in: placement. This one is usually a launcher configuration and it affects a
node consistently from the moment the job starts rather than appearing mid-run
4. host load and other processes (1 minute)
look for anything else consuming cores or memory bandwidth: a stuck monitoring agent, a
leftover process from a previous job, a filesystem client under load
rules out / in: contention. Distinctive because it affects the host-side stages of the step
rather than the GPU work
5. GPU health counters (1 minute)
memory error rates, pending remaps, and any recent fault codes
a degrading GPU can be slow before it fails, particularly when errors are being corrected
at a high rate
rules out / in: the device itself
Thermal, Power and Cooling Events covers the first check; NVLink and Fabric Faults covers the second; DCGM and GPU Telemetry has the field names for all of them.
The discriminator to run early, because it halves the search space:
what each branch means
slowness follows the node hardware or node configuration: the five checks apply
node is fine in another job the job put something specific on that node: an uneven
data shard, a rank-conditional code path, or the rank that
handles an extra responsibility such as checkpoint writing
another node becomes slow the job has a structural straggler, not a hardware one.
Common cause: rank 0 doing extra work, or a data
distribution where one shard is larger
sanity: the third branch is the one people miss. If the slow rank moves when you change nodes,
no amount of hardware investigation will find anything, and every check will pass
When everything checks clean:
this happens, and the usual explanations in order:
a marginal component that tests fine under diagnostic load and fails under the job's actual
pattern. Run the job's own workload on that node alone and compare against a peer
a firmware or driver difference on that one node, which a fleet-wide version audit finds
in one query
a power or cooling condition specific to its position in the rack, which correlates with
time of day or with neighbors' load
the escalation: drain the node, run the long diagnostic, and if it still passes, keep it out
of the largest jobs and watch whether it recurs. A node that is reproducibly slow and
reproducibly healthy is a node to retire rather than to keep investigating
The reversal condition: none of this applies if the slow rank is slow only during collectives while its compute is normal, and its neighbors on the same node are equally affected. That points at the node's network path rather than the node itself, and the investigation moves to its NIC, its cable and its switch port, where a link running at a degraded width produces exactly this pattern. The distinguishing measurement is compute time against collective time per rank, which the observability design should be collecting anyway.
What interviewers probe next
- "Why does the mean not show it?" Because 127 of 128 ranks are fine, so the average moves by less than a percent while the job loses half its speed. Per-rank distribution rather than aggregate is the fix.
- "What does throttling at half clock cost exactly?" Roughly proportional for compute-bound work, since the tensor cores run at the clock. Memory-bound phases degrade less, so a step that is a mix degrades somewhere between.
- "How do you catch this automatically?" Alert on the spread across ranks, maximum against median, rather than on any absolute step time.
- "What if two nodes are slow?" Look for what they share: a rack, a switch, a power distribution unit, a firmware version. Two is more informative than one because the intersection is small.
Common mistakes
- Investigating the node before testing whether the slowness follows the node or the job.
- Reading aggregate step time, which moves 0.7% when the job loses 49%.
- Checking GPU health first, which is the most involved check and the least often the answer.
- Continuing to investigate a node that is reproducibly slow and reproducibly healthy, rather than retiring it.
Key takeaways
- Collectives are barriers, so one node at 2x the step time costs the job 49% while the mean across 128 ranks moves 0.7%.
- Check in order of effort: clocks and throttle reasons, link counters, processor affinity, host load, GPU health.
- Run the follow-the-node test early: excluding the node separates hardware from a job-structural straggler in one restart.
- A node that is reproducibly slow and reproducibly healthy should be retired rather than investigated further.