You have 16,384 H100s and a 405B dense model. Choose the parallelism layout and justify the order of the axes.

Tensor parallel 8, pipeline 16, data parallel 128: the layout Meta trained Llama 3 405B on, rebuilt from the memory floor, the compute budget and the bandwidth of each link. Why the axes go in that order, what each one costs per step, and the bubble that the schedule has to hide.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🕸️ Distributed Training🔒 Premium

Tensor ParallelismTensor parallelism splits individual weight matrices across GPUs so each rank computes a slice of every layer, which is how a model whose single layer does not fit one GPU gets trained at all. It costs four all-reduces per transformer block on the critical path, which is why it stays inside the NVLink domain and rarely exceeds 8 ranks.

Advanced

🕸️ Distributed Training🔒 Premium

Pipeline Parallelism and the BubblePipeline parallelism puts consecutive groups of layers on different GPUs and streams micro-batches through them, which is the only parallelism whose traffic is small enough to cross a slow fabric comfortably. Its cost is the bubble, the idle time while the pipeline fills and drains, and the schedule you pick (GPipe, 1F1B, interleaved, zero-bubble) decides how much of each step is wasted.

Core

🕸️ Distributed TrainingSign in

ZeRO and FSDPZeRO and FSDP keep data parallelism's simple programming model but shard the optimizer state, gradients and parameters across ranks, cutting per-GPU memory from 16 bytes per parameter toward 16/N. The price is 1.5x DDP's communication and a dependence on tokens per GPU that decides when sharding stops paying and tensor parallelism takes over.

Foundational

🩺 Fleet Reliability & Observability

Training Uptime and Interruption StatisticsA frontier training run is a months-long job on tens of thousands of parts, each of which fails rarely, so the run as a whole fails constantly. The numbers that describe this are simple and worth deriving from scratch: a per-GPU failure rate, multiplied by the fleet, gives the interval between stops; the checkpoint interval and the restart time give the work lost per stop; together they give effective training time, which is the fraction of wall-clock the run is actually learning. This page derives all of it from the one public dataset that reports it and shows how to size checkpointing so the run stays above 90%.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackExplain mixed-precision training. Why does fp16 need loss scaling and bf16 not, why keep fp32 master weights, and what changes with fp8?Next in this trackYour training run is at 60% of the step time you projected. How do you find out whether it is compute, memory, network or I/O?Next in this trackTraining loss went flat at step 40k after descending normally. Walk me through how you debug it.

FEDITOR'S NOTE

Scored on deriving the layout from three constraints rather than reciting it, on ordering the axes by bandwidth need, and on doing the pipeline bubble arithmetic honestly.

DISCUSSION · 0

No comments yet — be the first to share your approach.