NVIDIA Networking, Interconnects & Storage interview questions

Before the first byte moves, the library has already discovered the machine's topology, built rings and trees over it, split the work into parallel channels and chosen a wire protocol by message size. What each of those four decisions is for, and the environment variables that let you see and override them.

Four operations cover almost everything a distributed training step sends. What each one does to the data, the bytes each rank moves for a message of size S across N ranks, and which parallelism strategy generates which, worked through for a 70B model so the numbers are concrete rather than symbolic.

Both carry RDMA at the same line rate. One arrives lossless because the transport was designed that way, the other becomes lossless only if a set of switch settings is correct on every port. What that difference costs in operations, what it saves in money and hiring, and the fleet size where the answer flips.

A single core moves a few gigabits per second of TCP once copies and interrupts are counted, so filling one 400 gigabit link would take most of a server's cores doing nothing but networking. What RDMA removes, what a queue pair actually is, and where the GPU fits when the data never belongs to the host at all.

Wire GPU number three of every node to the same switch and the collective that matters most never leaves it. The rank mapping that makes the layout pay, the switch and cable count for a thousand GPUs worked out, and what happens to a job whose ranks are assigned in the wrong order.

Thirty-two senders aimed at one port fill a switch buffer in microseconds, and what happens next depends on which of two mechanisms fires first. The buffer arithmetic, why the slow mechanism is the safe one and the fast one is the dangerous one, and the counters that tell you which is happening.

Reading straight from NVMe into GPU memory skips a copy through host memory, which is worth a lot for one workload and nothing for another. The bandwidth arithmetic for both paths, the two cases where the bounce is the bottleneck, and the far more common case where the CPU has to touch the data anyway.

Three parallelism dimensions, two link speeds eighteen times apart, and one mapping that makes the job fast. The traffic each dimension generates per step, why the loudest one must stay inside the node, and the arithmetic showing what a wrong rank order costs before anyone notices.

Four of them diagnose, four configure the machine correctly, and several popular ones override tuning that was already right. Which is which, the symptom each addresses, and why a cluster whose job scripts carry a dozen of these has a configuration problem rather than a tuning one.

The number comes from one requirement: the gradient reduction has to finish inside the backward pass that produces it. Working that requirement backward gives a bandwidth per GPU, and the answer lands near the port speed the industry ships, which is not a coincidence.

Two all-to-alls per layer across sixty layers is over a hundred collectives on the critical path of every decode step, each one small and latency-bound. The per-GPU volume worked out, the time on each link type, and why a single large NVLink domain changes the design rather than just improving it.

It fails open: the data still moves, the job still runs, and everything is about half as fast. The one log line that settles it, the four reasons the library declines to use it, and why the bandwidth signature is close enough to exactly half to be diagnostic on its own.

Find the scale at which it breaks before looking at any node, because the scale tells you which layer to open. The commands in order, the number each should return on a healthy fabric, and the four causes that this measurement sequence separates without guessing.

Port counts decide everything: a two-tier Clos of 64-port switches tops out at exactly 4,096 endpoints at full bisection, which is why that number appears in so many cluster designs. The switch and optics count both ways, what oversubscription actually slows down, and where the money goes.

One gives you POSIX semantics and hundreds of gigabytes per second at a price per terabyte that hurts at petabyte scale. The other is cheap and durable and answers a first byte in tens of milliseconds. The cost arithmetic that makes most clusters run both, and the metadata limit that decides how you shard.

For text training the bytes are trivial and the bottleneck is never bandwidth, so the usual advice about faster storage misses. What the loader actually has to deliver per second, the four things that consume the time instead, and the order to fix them with the measurement that proves each.

A tokenizer moves about a megabyte of text per second per core, which makes this a seven-hundred-core-day batch job rather than something to run during training. The throughput arithmetic per stage, the shard format the loader needs, and the determinism requirements that let you resume without corrupting a run.

Six and a half terabytes leaving sixteen thousand GPUs at once is a burst no shared filesystem absorbs, and the fix is to stop trying. What the barrier actually has to wait for, the per-node arithmetic that makes it seconds, and the manifest rule that decides whether a checkpoint is usable at all.