Advanced

Topology-Aware Communication

The same collective can run at 900 GB/s or at 50 GB/s depending on which links it is laid across, so the mapping of parallel groups onto hardware is a performance decision, not a deployment detail. The rule: tensor-parallel groups inside the NVLink domain, data-parallel rings along rails, pipeline stages across the fabric, and every rank placed so its partner is one hop away. NCCL discovers the topology and does most of this when the job lets it; the failures come from placements that do not.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

NVLink, NVSwitch and PCIe Rail-Optimized and Fat-Tree Fabrics Tensor Parallelism NCCL and Collective Algorithms Topology-Aware Scheduling Debugging a Slow All-Reduce

LESSONS THAT TEACH THIS

Fleet and ReliabilityWhere the devices sit matters more than which scheduler assigned them14 min→Inference EngineeringSplitting a model across devices buys memory and spends interconnect15 min→Training at ScaleMap the mesh so the chattiest dimension takes the shortest path14 min→

PRACTICE THIS IN REAL QUESTIONS

Networking, Interconnects & StorageMap a 1,024-GPU job's parallelism onto the hardware. Which dimension goes on NVLink, which on the fabric, and what does a wrong order cost?→Kubernetes, Slurm & GPU SchedulingTwo identical 8-GPU jobs get 8 GPUs each. One runs at half the speed of the other. What did the scheduler do, and how do you stop it?→GPU & Accelerator ArchitectureNVLink versus PCIe: when does the interconnect actually matter for an LLM workload, and when can you get away with PCIe?→Networking, Interconnects & StorageExplain a rail-optimized fabric. What does it buy over a plain fat tree, and how many switches does one need?→GPU & Accelerator ArchitectureExplain GPUDirect RDMA. What does it remove from the data path, and when does it change performance?→Open-Weights Models & Serving EnginesA multi-node vLLM deployment starts, forms a cluster, and hangs on the first request. Debug it.→