AI Infra Interviews logo
← Coding for Infra / 13
mediumNewNVIDIAGoogle

Simulate a ring all-reduce on arrays. Verify both the result and the bytes each rank sends.

Two phases of N-1 steps each, a chunk index that rotates with the step, and a byte count that should come out to exactly the textbook formula. The implementation, the verification against a direct sum, and the measured traffic matching 2(N-1)/N times the message at every rank count tested.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

Two phases of N-1 steps each, a chunk index that rotates with the step, and a byte count that should come out to exactly the textbook formula. The implementation, the verification against a direct sum, and the measured traffic matching 2(N-1)/N times the message at every rank count tested.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced
🕸️ Distributed Training🔒 Premium
Ring vs Tree All-ReduceA ring all-reduce moves the minimum possible bytes per rank but takes 2(N-1) steps, so its latency grows with the number of GPUs; a tree finishes in a logarithmic number of steps but is harder to keep bandwidth-optimal. NCCL keeps both, chooses per message size and rank count, and reading its choice is how you diagnose a collective that is slower than the fabric allows.
Foundational
🕸️ Distributed Training
Collective Communication PrimitivesAll-reduce, all-gather, reduce-scatter, all-to-all and broadcast are the five operations every parallelism strategy is built from, and each has a fixed per-rank traffic cost you can compute before a job runs. Knowing those volumes for a named model is how you decide whether a layout is compute-bound or waiting on the network.
Foundational
🔌 Networking & Storage
NCCL and Collective AlgorithmsNCCL is the library every PyTorch collective lands in, and its choice of ring or tree, channel count and protocol decides whether an all-reduce runs at fabric speed or at a third of it. Knowing what NCCL_DEBUG=INFO prints, and which environment variable changes which decision, is the difference between tuning a cluster and guessing at it.
Foundational
🖧 Hardware & Cluster Build-Out
NVLink Domains and the NVL72 RackAn NVLink domain is the set of GPUs that can address each other's memory at full fabric speed, and its size is the single most consequential number in a cluster design. Eight on an HGX node, 72 on a GB300 NVL72 rack. Inside the domain a collective moves at terabytes per second over a copper backplane; outside it, the same collective drops to the scale-out fabric at 800 Gb/s per GPU, a gap of roughly twenty times that decides how models are sharded.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackImplement the block allocator behind a paged KV cache, including prefix sharing and copy-on-write.Next in this trackImplement the scheduler loop of a continuous-batching engine: admit, step, preempt. Where does it livelock?Next in this trackIn Go, download many model shards concurrently with a concurrency limit, a timeout and clean cancellation. Write it.
FEDITOR'S NOTE

Scored on the two-phase structure with the rotating chunk index, on verifying against a direct sum rather than asserting correctness, and on measuring the traffic against the closed-form expression.

DISCUSSION · 0

No comments yet — be the first to share your approach.