← 🧮 Napkin Math & Capacity
Advanced
Communication Volume Estimates
Every parallelism strategy is a promise to move a certain number of bytes between GPUs every step, and the fabric either affords it or it does not. This page derives the per-rank volume for data parallelism, ZeRO/FSDP, tensor parallelism, pipeline parallelism and expert parallelism, works each for a 70B model at 8 and 64 ranks, and turns the bytes into seconds on NVLink and on a 400 Gb/s NIC. The result is the rule that decides every 3D layout: per-layer traffic stays on NVLink, per-step traffic can cross the fabric.
Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew
RELATED CONCEPTS
LESSONS THAT TEACH THIS
PRACTICE THIS IN REAL QUESTIONS
Networking, Interconnects & StorageName the four collectives a training job uses, say what each moves, and match them to the parallelism that needs them.→Networking, Interconnects & StorageMap a 1,024-GPU job's parallelism onto the hardware. Which dimension goes on NVLink, which on the fabric, and what does a wrong order cost?→Networking, Interconnects & StorageDoes multi-node inference need the same fabric as training, or can it run on ordinary networking?→Napkin Math, Cost & CapacityEstimate the latency of one decode step for a 70B model under tensor parallelism across eight H100s→Napkin Math, Cost & CapacityHow long does an all-reduce of a 70B model's gradients take on eight GPUs?→GPU & Accelerator ArchitectureNVLink versus PCIe: when does the interconnect actually matter for an LLM workload, and when can you get away with PCIe?→