Advanced

Disaggregated Prefill and Decode

Prefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Prefill vs Decode Chunked Prefill The KV Cache Inference Platform Architecture RDMA, InfiniBand and RoCEv2 Request Routing and Load Balancing for LLMs

LESSONS THAT TEACH THIS

Inference EngineeringA long prefill blocks every decode behind it, and chunking is the fix14 min→Inference EngineeringSplitting prefill from decode answers a problem you must have first15 min→

PRACTICE THIS IN REAL QUESTIONS

LLM Inference & ServingWhen does splitting prefill and decode onto separate GPU pools pay for itself, and what does the KV transfer cost?→LLM Inference & ServingWhy do prefill and decode behave so differently, and why does that matter for the hardware you serve on?→Open-Weights Models & Serving EnginesWould you split prefill and decode onto separate pools for a 753B model? Work the decision.→AI Infrastructure System DesignDesign prefill and decode disaggregation for a 70B serving fleet: pool sizes, KV transfer, routing, and when to turn it on.→LLM Inference & ServingA long prompt arrives while sixty users are mid-generation. What happens, and how does chunked prefill fix it?→LLM Inference & ServingDesign the fastest serving path for a 70B model with a 200 ms TTFT target at production scale.→