Networking, Interconnects & Storage: the practice test

NCCL implements the collectives distributed training needs, running directly between GPU memories over NVLink inside a node and RDMA across nodes. At initialization it discovers the topology (NVLink, PCIe, NICs, switches) and builds rings and trees over it; at each call it chooses ring, tree or other algorithms and the protocol by message size and the tuning tables, aiming to keep large collectives bandwidth-bound and small ones latency-bound. Its environment variables and debug output are the first tools for a slow all-reduce.

With remote direct memory access, an application registers a memory region, and the NIC moves bytes into a remote registered region without a system call per packet, a kernel copy, or CPU involvement on the data path. That is what lets a 400 Gb/s NIC deliver close to 50 GB/s with microsecond latency, and with GPUDirect the registered region can be GPU memory, so a collective runs GPU to GPU across nodes with the CPU idle. InfiniBand and RoCE v2 are the two transports that carry it.