Advanced

Chunked Prefill

A long prompt's prefill can occupy a GPU for hundreds of milliseconds, and every sequence mid-decode on that GPU waits for it. Chunked prefill splits the prompt into fixed token budgets and interleaves each chunk with a decode step, so decode latency stays flat at the cost of a slower first token for the long prompt. The chunk budget is a knob between TTFT and TPOT, and the interview question is how you would set it.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Prefill vs Decode Continuous Batching Latency Metrics: TTFT, TPOT and Goodput Disaggregated Prefill and Decode Designing for Latency SLOs Memory-Bound vs Compute-Bound Kernels

LESSONS THAT TEACH THIS

Foundations of AI InfrastructureA request becomes tokens, and tokens become two different workloads12 min→Foundations of AI InfrastructureBatching is a scheduling decision, not a configuration flag13 min→Inference EngineeringPrefix reuse is free capacity, and most fleets leave it on the floor14 min→

PRACTICE THIS IN REAL QUESTIONS

LLM Inference & ServingA long prompt arrives while sixty users are mid-generation. What happens, and how does chunked prefill fix it?→Open-Weights Models & Serving EnginesYour p99 time to first token is four times p50. Find out why.→LLM Inference & ServingWhy do prefill and decode behave so differently, and why does that matter for the hardware you serve on?→LLM Inference & ServingDesign the fastest serving path for a 70B model with a 200 ms TTFT target at production scale.→LLM Inference & ServingWhat is the difference between static and continuous batching, and why did it change LLM serving?→LLM Inference & ServingDefine TTFT, TPOT and goodput, and tell me how you would measure each one in production.→