TL;DR: Under plain continuous batching an admitted prompt is prefilled as one step, so a 20k-token prompt on a 70B model stalls every running stream for about 900 ms and the inter-token latency histogram goes bimodal. Chunked prefill gives each step a token budget (say 2,048 tokens) that prefill chunks fill after decode tokens are placed, so decode continues at a bounded step time while the long prompt's TTFT stretches from one step to ten. The budget is set from the TPOT SLO, and it reverses when TTFT on long prompts is the SLO that matters.
How to approach it
Describe the scenario in engine terms: the scheduler admits the request, and a prefill of N tokens must run before its first decode. Ask for the TPOT and TTFT SLOs, because the chunk size is derived from them. Then quantify the stall, explain why a mixed step bounds it, derive the chunk budget, and give the cost on the long prompt's TTFT. Close with the metric that shows the problem and the one that shows the fix.
A strong answer
A typical situation: p99 inter-token latency is fine most of the day and spikes to twenty times p50 without warning, and the users who complain are never the ones who sent the long prompt.
Take Llama 3.1 70B on 8 H100s, 60 sequences decoding at about 8.4 ms per step, and a 20k-token document arriving. Continuous Batching admits it on the next step, and the naive scheduler runs its prefill as a standalone step:
prefill time ≈ tokens × 2 × params ÷ (GPUs × peak × MFU)
= 20,000 × 1.41e11 ÷ (8 × 989e12 × 0.4)
= 2.82e15 ÷ 3.16e15 ≈ 890 ms
sanity: prefill is compute-bound, so this scales linearly with prompt length;
a 2k prompt would cost 89 ms, which is a visible but tolerable blip
For 890 ms none of the 60 streams receive a token. Their mean TPOT barely moves (one 890 ms gap in a 300-token answer adds 3 ms to the average), but the user sees a freeze, and the inter-token latency histogram grows a second mode near 900 ms aligned with long-prompt admissions. At p99 on the per-token gap, this is the whole tail.
Chunked Prefill changes the step composition. Each step gets a token budget. Decode tokens are placed first (60 tokens for 60 sequences), and prefill chunks fill the remainder. With a 2,048-token budget:
budget per step = 2,048 tokens
decode tokens per step = 60 (one per running sequence)
prefill tokens per step = 2,048 - 60 = 1,988
chunks for a 20,000-token prompt = ceil(20,000 ÷ 1,988) = 11 steps
mixed step time ≈ decode read time + chunk compute time
decode: (141.2 GB + 60 × 1.34 GB) ÷ 26.8 TB/s ≈ 8.3 ms
chunk: 1,988 × 1.41e11 ÷ 3.16e15 ≈ 89 ms
in practice the weight read overlaps the chunk compute, so the step is near max(8.3, 89) plus
the attention over the chunk, call it 90 to 100 ms
sanity: running streams see 11 steps of about 95 ms instead of one gap of 890 ms;
the long prompt's TTFT becomes 11 × 95 ≈ 1.05 s instead of 0.89 s
That is the trade. Decode TPOT during the prefill window is bounded near 95 ms rather than spiking to 890 ms; the long prompt's TTFT grows by about 20% because its chunks share steps with decode and the attention over earlier chunks is recomputed against the growing cache. The chunk budget sets where that lands. A 512-token budget holds the mixed step near 30 ms and stretches the long prompt to 40 steps and about 1.2 s. An 8,192 budget nearly recreates the stall.
Choosing the budget from the SLOs:
| TPOT p99 SLO | Budget that respects it (70B, 8 H100, MFU 0.4) | 20k prompt TTFT |
|---|---|---|
| 50 ms | about 1,000 tokens | about 1.1 s |
| 100 ms | about 2,048 tokens | about 1.0 s |
| 200 ms | about 4,096 tokens | about 0.95 s |
The reversal condition: a workload where every prompt is long and there is little decode (batch document processing) loses throughput to chunking because prefill efficiency drops for small chunks and the per-chunk attention recompute grows. There, prefill as whole steps, or a Disaggregated Prefill and Decode split so decode never shares a card with prefill at all.
The reversal condition: a workload of uniformly short prompts, where no single prefill is long enough to block anyone and the chunking machinery costs a little scheduling overhead for nothing. Measure the prompt-length distribution before turning it on.
The signals: before the fix, a bimodal inter-token latency histogram whose second mode tracks the prompt-length distribution; after it, a unimodal histogram with a raised shoulder near the mixed-step time, and a prefill queue-depth metric that shows how many chunks are pending.
What interviewers probe next
- "Why does prefill get slower when chunked?" Each chunk's attention runs over all earlier chunks' KV, so the attention term grows per chunk, and small chunks run the linear layers below peak MFU; total prefill cost rises 10 to 30% depending on chunk size.
- "Where does the decode token go in the budget?" First, always; the budget exists to protect TPOT, and vLLM's scheduler places running sequences before waiting ones so decode never starves.
- "What about a burst of ten long prompts?" They queue in the prefill lane and each gets its chunks in order; TTFT for the tenth is roughly ten prompts' worth of chunks, which is a queueing problem the router or a prefill pool must handle.
Common mistakes
- Quoting a default chunk size without deriving it from the SLO.
- Claiming chunked prefill is free for the long prompt.
- Confusing it with disaggregation, which moves prefill to another device rather than slicing it.
- Not knowing that the per-token mean hides the stall and only the histogram shows it.
Key takeaways
- A 20k-token prefill on a 70B model on 8 H100s at MFU 0.4 costs about 890 ms, and as one step it freezes every running stream.
- Chunking gives each step a token budget; decode tokens are placed first, prefill chunks fill the rest.
- Budget follows from the TPOT SLO: about 2,048 tokens for a 100 ms p99 on this fleet.
- Cost is 10 to 30% more prefill compute and a longer TTFT for the long prompt; the ITL histogram is the proof either way.