CoreWeave Networking, Interconnects & Storage interview questions

Wire GPU number three of every node to the same switch and the collective that matters most never leaves it. The rank mapping that makes the layout pay, the switch and cable count for a thousand GPUs worked out, and what happens to a job whose ranks are assigned in the wrong order.

Find the scale at which it breaks before looking at any node, because the scale tells you which layer to open. The commands in order, the number each should return on a healthy fabric, and the four causes that this measurement sequence separates without guessing.

Four of them diagnose, four configure the machine correctly, and several popular ones override tuning that was already right. Which is which, the symptom each addresses, and why a cluster whose job scripts carry a dozen of these has a configuration problem rather than a tuning one.

Three networks, not one, and the mistake that hurts most is putting two of them on the same wires. The port and switch counts for each, the cable plan, the split of the bill between switches and optics, and the two decisions that are hard to change once the cluster is racked.

A collective stops when one participant stops, and a job of a thousand ranks has no way to distinguish a link that will come back in two seconds from a rank that has died. What the job actually does during those seconds, the cost per flap, and why the fault passes every test you would run.

Every tenant bursts independently, so the shared tier sees the sum of uncorrelated peaks rather than the sum of averages. The provisioning arithmetic that follows, the three isolation boundaries, and why per-tenant local storage solves most of it before any policy is written.

A uniform loss across every node rules out hardware and points at something that changed everywhere at once. The differential procedure that isolates a reset default from a real regression, the settings firmware resets on its own, and the change process that catches it in an hour.

Before the first byte moves, the library has already discovered the machine's topology, built rings and trees over it, split the work into parallel channels and chosen a wire protocol by message size. What each of those four decisions is for, and the environment variables that let you see and override them.

Four operations cover almost everything a distributed training step sends. What each one does to the data, the bytes each rank moves for a message of size S across N ranks, and which parallelism strategy generates which, worked through for a 70B model so the numbers are concrete rather than symbolic.

Both carry RDMA at the same line rate. One arrives lossless because the transport was designed that way, the other becomes lossless only if a set of switch settings is correct on every port. What that difference costs in operations, what it saves in money and hiring, and the fleet size where the answer flips.

A single core moves a few gigabits per second of TCP once copies and interrupts are counted, so filling one 400 gigabit link would take most of a server's cores doing nothing but networking. What RDMA removes, what a queue pair actually is, and where the GPU fits when the data never belongs to the host at all.

Port counts decide everything: a two-tier Clos of 64-port switches tops out at exactly 4,096 endpoints at full bisection, which is why that number appears in so many cluster designs. The switch and optics count both ways, what oversubscription actually slows down, and where the money goes.

Thirty-two senders aimed at one port fill a switch buffer in microseconds, and what happens next depends on which of two mechanisms fires first. The buffer arithmetic, why the slow mechanism is the safe one and the fast one is the dangerous one, and the counters that tell you which is happening.

Reading straight from NVMe into GPU memory skips a copy through host memory, which is worth a lot for one workload and nothing for another. The bandwidth arithmetic for both paths, the two cases where the bounce is the bottleneck, and the far more common case where the CPU has to touch the data anyway.

One gives you POSIX semantics and hundreds of gigabytes per second at a price per terabyte that hurts at petabyte scale. The other is cheap and durable and answers a first byte in tens of milliseconds. The cost arithmetic that makes most clusters run both, and the metadata limit that decides how you shard.

For text training the bytes are trivial and the bottleneck is never bandwidth, so the usual advice about faster storage misses. What the loader actually has to deliver per second, the four things that consume the time instead, and the order to fix them with the measurement that proves each.

A tokenizer moves about a megabyte of text per second per core, which makes this a seven-hundred-core-day batch job rather than something to run during training. The throughput arithmetic per stage, the shard format the loader needs, and the determinism requirements that let you resume without corrupting a run.

Six and a half terabytes leaving sixteen thousand GPUs at once is a burst no shared filesystem absorbs, and the fix is to stop trying. What the barrier actually has to wait for, the per-node arithmetic that makes it seconds, and the manifest rule that decides whether a checkpoint is usable at all.

Three parallelism dimensions, two link speeds eighteen times apart, and one mapping that makes the job fast. The traffic each dimension generates per step, why the loudest one must stay inside the node, and the arithmetic showing what a wrong rank order costs before anyone notices.

Three tools measure three different things, and a link can pass one while failing another. The message-size sweep that separates a latency problem from a bandwidth problem, the number each test should return on a healthy 400 gigabit port, and the single test that catches the failure the others miss.

The number comes from one requirement: the gradient reduction has to finish inside the backward pass that produces it. Working that requirement backward gives a bandwidth per GPU, and the answer lands near the port speed the industry ships, which is not a coincidence.

RoCE inherited a transport that assumes one ordered path per connection, which is why it needs a lossless network built by hand underneath it. What a redesigned transport changes about that, which of today's operational burdens it removes, and why the honest answer to a purchase decision is still about what ships.