TL;DR: 100e9 × 16 B = 1.6 TB of static training state, which is 2.5 nodes of 8 × H100 before activations, so the model spans nodes and the question is how. Tensor parallelism of 8 inside each node over NVLink, then either FSDP across nodes (simpler, about 3 model-sizes of traffic per step) or pipeline parallelism across nodes (less traffic, a bubble), with data parallelism over whatever remains. The fleet size comes from compute, not memory: 2 trillion tokens at 40% MFU on 1,024 H100s is about 34 days.
How to approach it
Do the memory floor out loud first: bytes per parameter times parameters, compared to a node. Then say that memory only sets a minimum and compute sets the real fleet, and size the fleet from a token budget and a deadline. With the fleet size in hand, assign parallelism axes to links: TP on NVLink, the cross-node axis on the NIC, and DP across the rest. State the FSDP-vs-PP decision as a comparison of per-step bytes against per-step compute. Close with the layout as a product of three numbers and the per-rank memory it leaves.
A strong answer
A typical situation: a lab has 1,024 H100s for six weeks and a 100B dense architecture with 2 trillion tokens of data. The layout has to satisfy memory, compute and network at once, so take them in that order.
1. memory floor
static state = 100e9 × 16 B = 1.6e12 B = 1,600 GB
one node = 8 × 80 GB = 640 GB
minimum GPUs to hold state at 80% usable = 1,600 ÷ (80 × 0.8) = 25 → 32 GPUs (4 nodes)
sanity: 4 nodes hold the state and nothing else; activations come on top,
so the memory floor is a lower bound, not a plan
2. compute sets the fleet
FLOPs = 6 × 100e9 × 2e12 = 1.2e24
on 1,024 H100s at 989 TFLOPS and MFU 0.4:
rate = 1,024 × 989e12 × 0.4 = 4.05e17 FLOP/s
time = 1.2e24 ÷ 4.05e17 = 2.96e6 s ≈ 34 days
sanity: on the 32 GPUs from step 1 the same run takes 1,090 days; memory told us
where the floor is, compute told us the fleet is 32× larger than the floor
3. per-rank memory at 1,024 GPUs
static state per GPU if fully sharded = 1,600 GB ÷ 1,024 = 1.6 GB
sanity: the state is now trivial per GPU; what fills the 80 GB is activations,
communication buffers and the temporary gathered weights of one layer
With 1,024 GPUs, the layout question is which axes to use, not whether it fits. Two designs are reasonable.
Design A: TP8 × FSDP128. Tensor Parallelism across the 8 GPUs of each node splits every layer's matrices, so each GPU computes 1/8 of every GEMM and the four activation all-reduces per block run on NVLink at 900 GB/s. Across the 128 nodes, FSDP shards the TP-sharded parameters further and gathers them per layer. Traffic per step across the NIC is about 3 × the per-node parameter bytes:
params per TP rank = 100e9 ÷ 8 = 12.5e9 → 25 GB in bf16
FSDP traffic per rank per step ≈ 3 × 25 GB = 75 GB → at 50 GB/s: 1.5 s
compute per rank per step, 16k tokens per DP replica:
6 × 100e9 × 16,384 ÷ 8 TP ranks = 1.23e15 FLOPs ÷ (989e12 × 0.4) ≈ 3.1 s
ratio ≈ 0.48: hidden if prefetch works, exposed if the micro-batch shrinks
Design B: TP8 × PP8 × DP16. Pipeline Parallelism and the Bubble splits the layers into 8 stages, one per node, so each node holds one eighth of the model and the FSDP all-gathers disappear; cross-node traffic is a 134 MB activation per micro-batch per boundary plus the once-per-step gradient all-reduce over the 16 DP replicas. The cost is the bubble, (p−1)/m of compute time: with 8 stages and 32 micro-batches that is 7/32 ≈ 22% of compute idle, and more micro-batches to shrink it means more activation memory held in flight.
| A: TP8 × FSDP128 | B: TP8 × PP8 × DP16 | |
|---|---|---|
| Cross-node bytes per step per GPU | about 75 GB (weights, twice, plus grads) | about 3 GB (activations plus a sharded grad all-reduce) |
| Idle from structure | none | bubble 7/(m) of compute |
| Micro-batch needed | large enough to hide 1.5 s | many small ones to shrink the bubble |
| Code | one fully_shard mesh | stage assignment, schedule, balance |
| Failure of a node | reshard across 127 nodes | stage is gone, whole pipeline replica stalls |
Decision: Design A on 1,024 GPUs with 400 Gbps NICs, because a 0.48 communication-to-compute ratio hides behind one-layer-ahead prefetch and the code is a single device mesh. The condition that reverses it is bandwidth: on a 200 Gbps fabric the ratio doubles to about 1 and the run exposes communication on every step, at which point Design B's bubble of about 20% is cheaper than Design A's exposed 50%. Above roughly 4,000 GPUs, the FSDP all-gather over thousands of remote ranks also becomes latency-bound, and the Llama 3 answer, TP inside, PP across, DP over the rest, becomes the default.
One more number closes the memory question at the chosen layout: activations. At 16k tokens per micro-batch, hidden size around 12,288 for a 100B, about 34 × tokens × hidden bytes per layer without checkpointing is 6.7 GB per layer per GPU before TP's division; with TP8 and selective activation checkpointing this lands in the 30 to 50 GB range per GPU for a 100-layer model, which is why the 80 GB card is full even though the sharded state is 1.6 GB.
The reversal condition: a fabric that cannot carry the pipeline's activation sends inside the compute time, which pushes you back toward more tensor parallelism inside the node and fewer pipeline stages. Communication Volume Estimates is where those bytes come from. NCCL_DEBUG=INFO on the first run confirms the rings match the layout you drew.
What interviewers probe next
- "Why not TP16 across two nodes?" The four all-reduces per block would cross the NIC at 50 GB/s instead of NVLink at 900 GB/s; the activation traffic of about 75 GB per micro-batch would take 1.5 s on the NIC against 3 s of compute, and TP's all-reduce sits on the critical path with no overlap.
- "What if it were a 100B MoE with 20B active?" Memory floor is the same (1.6 TB, total parameters), compute drops 5×, and expert parallelism replaces most of TP; the all-to-all becomes the traffic to size.
- "How does the fleet change for 10 trillion tokens?" Linearly: 5× the FLOPs is 170 days on 1,024 GPUs or 34 days on 5,120, and at 5,120 the layout changes to include pipeline parallelism.
- "Where do checkpoints land in this?" 1.6 TB per full checkpoint, written sharded by every rank in parallel; at 2 TB/s aggregate that is under a second of I/O and the cost is the barrier, not the bytes.
Common mistakes
- Sizing the fleet from memory: "1.6 TB, so 32 GPUs" and stopping, which produces a three-year run.
- Forgetting the 16 and using 2 bytes per parameter (the inference number), then concluding the model fits on three cards.
- Proposing tensor parallelism across nodes because the model "needs more than 8 ways".
- Leaving activations out of the per-rank memory and being surprised that a 1.6 GB sharded state runs out of memory.
Key takeaways
- Memory floor: 100e9 × 16 B = 1.6 TB, 2.5 nodes; it is a minimum, not a plan.
- Compute sets the fleet: 6 × N × tokens ÷ (GPUs × peak × MFU); 2T tokens on 1,024 H100s at 40% is 34 days.
- TP8 inside the node on NVLink; across nodes, FSDP when cross-node traffic per step is under about half the compute time, PP when it is not.
- After sharding, activations fill the card, not parameters.