Why do large training clusters provision roughly 400 gigabits per second per GPU rather than more or less?

The number comes from one requirement: the gradient reduction has to finish inside the backward pass that produces it. Working that requirement backward gives a bandwidth per GPU, and the answer lands near the port speed the industry ships, which is not a coincidence.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL is the library every PyTorch collective lands in, and its choice of ring or tree, channel count and protocol decides whether an all-reduce runs at fabric speed or at a third of it. Knowing what NCCL_DEBUG=INFO prints, and which environment variable changes which decision, is the difference between tuning a cluster and guessing at it.

Foundational

🔌 Networking & Storage

Debugging a Slow All-ReduceA training job reports its all-reduce at a third of what the fabric should deliver, every node passed its health check, and nothing is logged. This page is the isolation order that finds the cause in an hour instead of a day: measure the collective in isolation, split the job until the slow pair or rank appears, then check the specific things that make a link, a node or a placement slow. Most cases end at one NIC, one topology mismatch, or GPUDirect silently off.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Capacity Planning and UtilizationCapacity planning for GPUs is deciding how many to have next quarter given that they cost money whether busy or not, that demand arrives in bursts, and that a queue near saturation produces waits that grow without bound. This page works the planning arithmetic for a serving fleet (peak demand, headroom, the p99 penalty of running hot) and a training platform (job mix, queue wait, the value of a shared pool), and gives the queueing intuition that makes 70% look full. The number that decides everything is utilization, and it has a ceiling set by latency, not by hardware.

Advanced

🕸️ Distributed Training🔒 Premium

Tensor ParallelismTensor parallelism splits individual weight matrices across GPUs so each rank computes a slice of every layer, which is how a model whose single layer does not fit one GPU gets trained at all. It costs four all-reduces per transformer block on the critical path, which is why it stays inside the NVLink domain and rarely exceeds 8 ranks.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhat is Ultra Ethernet trying to fix, and does it change how you would choose a fabric today?Next in this trackDesign the complete network for a 1,024-GPU cluster: compute fabric, storage, management. What are the counts and where does the money go?Next in this trackDoes multi-node inference need the same fabric as training, or can it run on ordinary networking?

FEDITOR'S NOTE

Scored on deriving the bandwidth requirement from overlap rather than quoting the number, on knowing which parallelism dimension sets it, and on how the answer changes with model size and generation.

DISCUSSION · 0

No comments yet — be the first to share your approach.