Several unrelated jobs slow down at once and nothing is dropping packets. How do you confirm a pause cascade and find where it started?

The signature is unusual: many jobs degrade together, throughput collapses, and the drop counters stay at zero. Reading pause counters backward along the paths to find the origin, the three conditions that produce one, and the configuration change that stops it happening again.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core

🔌 Networking & StorageSign in

RDMA, InfiniBand and RoCEv2Training across nodes moves hundreds of gigabytes per step, and a CPU-driven TCP stack cannot feed a 400 Gb/s link. RDMA lets a NIC write straight into a remote GPU's memory with no kernel and no copies, and it runs over two fabrics: InfiniBand, which is lossless by design, and RoCEv2, which is Ethernet made lossless by configuration. The choice is operational as much as technical, and the numbers that decide it are per-GPU bandwidth, the collective's volume, and who will debug a pause storm at 3 a.m.

Advanced

🔌 Networking & Storage🔒 Premium

Congestion Control for AI FabricsCollective traffic is the worst case a network can see: hundreds of senders transmit to the same receiver at the same instant (incast), every flow is large and long-lived, and RDMA cannot tolerate a dropped packet. Congestion control is the set of mechanisms (PFC, ECN with DCQCN, adaptive routing, packet spraying) that keep queues from overflowing without stalling the fabric. On plain Ethernet a busy all-reduce can fall to about 60% of link rate; with a tuned control loop it holds above 90%. Reading the counters that show which one you have is the on-call skill.

Foundational

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL is the library every PyTorch collective lands in, and its choice of ring or tree, channel count and protocol decides whether an all-reduce runs at fabric speed or at a third of it. Knowing what NCCL_DEBUG=INFO prints, and which environment variable changes which decision, is the difference between tuning a cluster and guessing at it.

Advanced

🔌 Networking & Storage🔒 Premium

Rail-Optimized and Fat-Tree FabricsA GPU cluster's network is built from two ideas: a fat tree (Clos) that gives every node a path to every other node with a chosen amount of oversubscription, and rail optimization, which wires GPU i of every node to the same leaf switch so the collectives that dominate training stay one hop away. Sizing one is arithmetic on port counts, and the interview question is usually that arithmetic: how many switches, what oversubscription, and where the NVLink domain ends and the fabric begins.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this track8,192 GPUs exceeds what a two-tier Clos of 64-port switches supports. Compare adding a tier against using higher-radix switches.Next in this trackHow would you know GPUDirect RDMA is not being used, given that nothing reports an error when it is disabled?Next in this trackDesign storage for a multi-tenant GPU cloud. How do you keep one customer's checkpoint burst from slowing another's training run?

FEDITOR'S NOTE

Scored on recognizing the signature (many jobs slow, zero drops, pauses climbing), on tracing pauses upstream to the origin rather than reading them at the victim, and on the fix being ECN thresholds rather than disabling pause.

DISCUSSION · 0

No comments yet — be the first to share your approach.