Does multi-node inference need the same fabric as training, or can it run on ordinary networking?

It depends entirely on which dimension you split the model along, and the three choices differ by two orders of magnitude in what they demand. The per-token traffic for each, the arithmetic that puts tensor parallelism on the fast link and pipeline parallelism anywhere, and where disaggregated serving lands.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🧮 Open Weights & Serving Engines

Multi-Node Serving TopologiesOnce a model needs more GPUs than one NVLink domain holds, the deployment shape becomes a real design decision. Tensor parallelism stays inside the node because it communicates twice per layer per token. Across nodes the choices are data parallelism with replicas, pipeline parallelism with a bubble, expert parallelism with an all-to-all, or disaggregation that runs prefill and decode on separate pools and ships the KV cache between them. Each has a different failure mode and a different scaling story.

Advanced

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodePrefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Communication Volume EstimatesEvery parallelism strategy is a promise to move a certain number of bytes between GPUs every step, and the fabric either affords it or it does not. This page derives the per-rank volume for data parallelism, ZeRO/FSDP, tensor parallelism, pipeline parallelism and expert parallelism, works each for a 70B model at 8 and 64 ranks, and turns the bytes into seconds on NVLink and on a 400 Gb/s NIC. The result is the rule that decides every 3D layout: per-layer traffic stays on NVLink, per-step traffic can cross the fabric.

Foundational

🔌 Networking & Storage

Dataset Lifecycle: Ingest, Shard and RetainA training dataset is not a file, it is a pipeline with four stages and a retention policy, and each stage has a different bottleneck. Ingest is metadata-bound rather than bandwidth-bound. Tokenization is CPU work that should happen once offline rather than every epoch. Sharding decides whether the training read is a stream or a storm of small files. And retention decides how much of the bill is paid for bytes nobody reads.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackOne fabric link goes down and comes back once an hour. What does that do to a 1,024-GPU training job, and how would you find it?Next in this trackWhy does object storage behave badly for random reads, and what does a training pipeline do about it?Next in this trackDeduplicate and quality-filter a multi-petabyte web corpus. What does that pipeline cost and where does it bottleneck?

FEDITOR'S NOTE

Scored on separating the parallelism choices by their per-token traffic, on the decode-step arithmetic showing why cross-node tensor parallelism is punishing, and on placing KV transfer for disaggregation correctly.

DISCUSSION · 0

No comments yet — be the first to share your approach.