hardNewNVIDIACoreWeaveMicrosoft

What actually changes with Blackwell and the NVL72 rack, and what does it do to how you would serve a large MoE model?

Three things change at once: dense compute more than doubles with an fp4 tensor core, HBM3e reaches 8 TB/s, and NVLink stops at 72 GPUs instead of 8. Work through what each does to tensor-parallel degree, per-step weight reads and expert placement for a 671B-parameter mixture of experts.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

GPU Generations: A100 to BlackwellFour NVIDIA generations are in fleets at once, and interviewers ask what each one changed, not what it is called. A100 to H100 added fp8 and tripled compute; H200 kept the die and grew memory; B200 doubled everything and added fp4; B300 stacked more HBM and cut fp64. This page carries the dense numbers for each, what they did to training and serving, and the marketing traps (sparse peaks, 192 versus 180 GB, die counting) that trip candidates. Dated September 2026.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

NVLink, NVSwitch and PCIeInside a node, GPUs talk over NVLink at 900 GB/s per H100 through an NVSwitch fabric that gives all eight cards full bandwidth to each other; to the host and to anything outside the node they talk over PCIe at 64 GB/s or a 400 Gb/s NIC at 50 GB/s. That fifteen-fold gap is why tensor parallelism stays inside the eight-GPU domain, why NVL72 changes the serving math for MoE, and why the question "how many GPUs share an NVLink domain?" is the first thing to ask about any cluster.

Foundational

🧮 Open Weights & Serving Engines

Expert Parallel and All-to-All BackendsA mixture-of-experts model can be split two ways and the choice changes everything. Tensor parallelism shards each expert across GPUs, which keeps every GPU busy and reads every expert's shard on every token. Expert parallelism gives whole experts to whole GPUs, which reads only the selected experts but requires an all-to-all to route tokens to them and back. The all-to-all is the cost, its backend is a configuration choice matched to the interconnect, and expert load imbalance is what actually limits the result.

Foundational

🖧 Hardware & Cluster Build-Out

NVLink Domains and the NVL72 RackAn NVLink domain is the set of GPUs that can address each other's memory at full fabric speed, and its size is the single most consequential number in a cluster design. Eight on an HGX node, 72 on a GB300 NVL72 rack. Inside the domain a collective moves at terabytes per second over a copper backplane; outside it, the same collective drops to the scale-out fabric at 800 Gb/s per GPU, a gap of roughly twenty times that decides how models are sharded.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYou have trained on GPUs. What transfers to training on TPUs, and what do you have to relearn?Next in this trackExplain how a systolic array computes a matrix multiply. Why is it efficient, and why does it want big square tiles?Next in this trackWould you move a cost-sensitive training and serving fleet from H100 to Trainium2? What do you gain, and what do you have to plan for?

FEDITOR'S NOTE

Scored on separating the three changes, quantifying each against H100 with the datasheet, and reasoning about MoE serving as an expert-placement problem that the 72-GPU domain changes qualitatively.

DISCUSSION · 0

No comments yet — be the first to share your approach.