Would you split prefill and decode onto separate pools for a 753B model? Work the decision.

The two phases want opposite machines, so running them together makes every setting a compromise. What splitting buys, the KV transfer that couples the pools, and the arithmetic that says whether the coupling is affordable for this model.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodePrefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.

Foundational

🧮 Open Weights & Serving Engines

Multi-Node Serving TopologiesOnce a model needs more GPUs than one NVLink domain holds, the deployment shape becomes a real design decision. Tensor parallelism stays inside the node because it communicates twice per layer per token. Across nodes the choices are data parallelism with replicas, pipeline parallelism with a bubble, expert parallelism with an all-to-all, or disaggregation that runs prefill and decode on separate pools and ships the KV cache between them. Each has a different failure mode and a different scaling story.

Foundational

🚀 Inference & Serving

Prefill vs DecodeAn LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

Memory-Bound vs Compute-Bound KernelsEvery kernel is limited by one of two walls: how fast bytes arrive from HBM, or how fast the tensor cores can multiply. Which wall applies is decided by arithmetic intensity against the ridge point, and the two regimes need opposite fixes. Decode, LayerNorm and softmax are memory-bound; prefill GEMMs are compute-bound; the interview question is which one you are looking at and what you would do about it.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackA multi-node vLLM deployment starts, forms a cluster, and hangs on the first request. Debug it.Next in this trackYour product forecasts 12,800 output tokens per second at peak. Size the fleet.Next in this trackIs speculative decoding worth enabling on a trillion-parameter mixture-of-experts model?

FEDITOR'S NOTE

Scored on the two phases having different bottlenecks, on computing the KV transfer cost, and on the operational complexity that has to be justified by the gain.

DISCUSSION · 0

No comments yet — be the first to share your approach.