Map a 1,024-GPU job's parallelism onto the hardware. Which dimension goes on NVLink, which on the fabric, and what does a wrong order cost?

Three parallelism dimensions, two link speeds eighteen times apart, and one mapping that makes the job fast. The traffic each dimension generates per step, why the loudest one must stay inside the node, and the arithmetic showing what a wrong rank order costs before anyone notices.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🔌 Networking & Storage🔒 Premium

Topology-Aware CommunicationThe same collective can run at 900 GB/s or at 50 GB/s depending on which links it is laid across, so the mapping of parallel groups onto hardware is a performance decision, not a deployment detail. The rule: tensor-parallel groups inside the NVLink domain, data-parallel rings along rails, pipeline stages across the fabric, and every rank placed so its partner is one hop away. NCCL discovers the topology and does most of this when the job lets it; the failures come from placements that do not.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

NVLink, NVSwitch and PCIeInside a node, GPUs talk over NVLink at 900 GB/s per H100 through an NVSwitch fabric that gives all eight cards full bandwidth to each other; to the host and to anything outside the node they talk over PCIe at 64 GB/s or a 400 Gb/s NIC at 50 GB/s. That fifteen-fold gap is why tensor parallelism stays inside the eight-GPU domain, why NVL72 changes the serving math for MoE, and why the question "how many GPUs share an NVLink domain?" is the first thing to ask about any cluster.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Communication Volume EstimatesEvery parallelism strategy is a promise to move a certain number of bytes between GPUs every step, and the fabric either affords it or it does not. This page derives the per-rank volume for data parallelism, ZeRO/FSDP, tensor parallelism, pipeline parallelism and expert parallelism, works each for a 70B model at 8 and 64 ranks, and turns the bytes into seconds on NVLink and on a 400 Gb/s NIC. The result is the rule that decides every 3D layout: per-layer traffic stays on NVLink, per-step traffic can cross the fabric.

Foundational

🧮 Open Weights & Serving Engines

Multi-Node Serving TopologiesOnce a model needs more GPUs than one NVLink domain holds, the deployment shape becomes a real design decision. Tensor parallelism stays inside the node because it communicates twice per layer per token. Across nodes the choices are data parallelism with replicas, pipeline parallelism with a bubble, expert parallelism with an all-to-all, or disaggregation that runs prefill and decode on separate pools and ships the KV cache between them. Each has a different failure mode and a different scaling story.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhich NCCL environment variables are worth knowing, what does each change, and which should you almost never set?Next in this trackHow do you establish that the link between two GPU nodes is healthy, and what numbers should each test return?Next in this trackWhy do large training clusters provision roughly 400 gigabits per second per GPU rather than more or less?

FEDITOR'S NOTE

Scored on ranking the dimensions by traffic per step and placing them accordingly, on the NVLink versus fabric bandwidth ratio, and on quantifying the penalty of a wrong mapping rather than calling it suboptimal.

DISCUSSION · 0

No comments yet — be the first to share your approach.