AI Infra Interviews logo
GPU Fleet Reliability & Observability / 20
hardNewMetaNVIDIA

Walk me through reading a collective flight-recorder dump. What is in it, and how do you find the rank that caused a hang?

Each rank keeps a ring buffer of its recent collectives, and on a timeout every rank dumps it. Comparing sequence numbers across a thousand of those buffers finds the missing participant in one pass, and the three patterns those comparisons produce point at three different causes.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

Each rank keeps a ring buffer of its recent collectives, and on a timeout every rank dumps it. Comparing sequence numbers across a thousand of those buffers finds the missing participant in one pass, and the three patterns those comparisons produce point at three different causes.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced
🩺 Fleet Reliability & Observability🔒 Premium
Stragglers and HangsSynchronous training runs at the speed of its slowest rank, so one GPU that is 30% slow makes a thousand GPUs 30% slow, and one rank that never arrives at a collective makes the other 1,023 wait in silence until a watchdog fires ten minutes later. Finding the slow rank and the stuck rank is the most common on-call task on a training fleet, and the tooling for it (per-rank timing, the NCCL flight recorder, stack dumps across ranks) is specific and learnable. This page derives the straggler tax from first principles, lists the causes in the order they actually occur, and gives the procedure for a hang.
Foundational
🕸️ Distributed Training
Collective Communication PrimitivesAll-reduce, all-gather, reduce-scatter, all-to-all and broadcast are the five operations every parallelism strategy is built from, and each has a fixed per-rank traffic cost you can compute before a job runs. Knowing those volumes for a named model is how you decide whether a layout is compute-bound or waiting on the network.
Foundational
🔌 Networking & Storage
NCCL and Collective AlgorithmsNCCL is the library every PyTorch collective lands in, and its choice of ring or tree, channel count and protocol decides whether an all-reduce runs at fabric speed or at a third of it. Knowing what NCCL_DEBUG=INFO prints, and which environment variable changes which decision, is the difference between tuning a cluster and guessing at it.
Foundational
🖧 Hardware & Cluster Build-Out
NVLink Domains and the NVL72 RackAn NVLink domain is the set of GPUs that can address each other's memory at full fabric speed, and its size is the single most consequential number in a cluster design. Eight on an HGX node, 72 on a GB300 NVL72 rack. Inside the domain a collective moves at terabytes per second over a copper backplane; outside it, the same collective drops to the scale-out fabric at 800 Gb/s per GPU, a gap of roughly twenty times that decides how models are sharded.
UP NEXT ON YOUR JOURNEY
FEDITOR'S NOTE

Scored on the record's fields and what each is for, on the comparison across ranks rather than reading one buffer, and on the three patterns (one behind, mismatched operation, all equal) mapping to distinct causes.

DISCUSSION · 0

No comments yet — be the first to share your approach.