TL;DR: At low batch a decode step reads all the weights to produce one token, so verifying five candidate tokens in one step costs almost the same as producing one. A draft model with acceptance rate 0.8 and four draft tokens yields about 3.4 accepted tokens per verify step, which is near 2x on batch-1 latency for a 70B model. The gain shrinks as batch grows because the verify step's extra tokens push the GPU toward the compute ridge (about 295 FLOP/B on H100), and past roughly batch 60 to 100 the draft overhead exceeds what verification saves.
How to approach it
Ask about batch size and the target latency before anything else, because speculation is a low-batch tool. Then lay out the mechanism in two sentences, derive expected accepted tokens from the acceptance rate, cost one speculation round against one plain step using the bandwidth math, and show where the batch dimension kills it. Name the draft options only after the numbers, and say which acceptance rate you would need to measure before committing.
A strong answer
A typical situation: speculative decoding is switched on fleet-wide because it helped in a demo, and aggregate throughput falls. The demo was one user at batch 1; the fleet runs at batch 64, where the spare compute it needs does not exist.
Speculative Decoding runs a cheap draft to propose k tokens, then runs the target model once over all k+1 positions and accepts the longest prefix whose tokens the target would have sampled (rejection sampling keeps the output distribution exactly that of the target). The reason verification is cheap is the same reason batch-1 decode is slow: the step is a weight read, and adding a few tokens to the step adds compute the GPU had to spare.
inputs: target Llama 3.1 70B bf16 on 8 × H100 (26.8 TB/s aggregate), batch 1
draft Llama 3.1 8B bf16 (16.1 GB), k = 4 draft tokens, acceptance rate α = 0.8
plain step time = 141.2 GB ÷ 26.8 TB/s ≈ 5.3 ms → 190 tokens/s
expected accepted tokens per round = (1 - α^(k+1)) ÷ (1 - α)
α = 0.8, k = 4: (1 - 0.8^5) ÷ 0.2 = (1 - 0.328) ÷ 0.2 ≈ 3.36 tokens
round time = k draft steps + 1 verify step
draft step ≈ 16.1 GB ÷ 26.8 TB/s ≈ 0.6 ms, but a small model on eight cards is latency-bound,
so budget 1.0 ms per draft step → 4.0 ms
verify step over 5 tokens: intensity 5 FLOP/B, still far below the ridge → ≈ 5.3 ms
round ≈ 9.3 ms for 3.36 tokens → 361 tokens/s
speedup ≈ 361 ÷ 190 ≈ 1.9x on per-stream latency
sanity: 3.36 tokens for the price of 1.75 plain steps; this is the range vendors report for
well-matched draft pairs on chat text
The acceptance rate is everything. Rerun at α = 0.5 and the expected accepted count is (1 - 0.5^5) ÷ 0.5 ≈ 1.94 tokens per 9.3 ms, which is 209 tokens/s, or 1.1x, barely covering the engineering. Code and structured output typically draft well (high α); creative text with a mismatched draft does not. You measure α on your traffic before deciding; the number is a property of the draft-target pair and the domain, not of the technique.
Now the batch dimension. At batch B the verify step processes B × (k+1) tokens. Decode intensity in bf16 equals tokens per step, so:
verify-step intensity = B × (k + 1) FLOP/B; H100 ridge ≈ 295
B = 1: 5 → memory-bound, verify ≈ plain step
B = 32: 160 → still memory-bound, but KV reads for 32 × 5 positions add up
B = 64: 320 → at the ridge; verify now costs (k+1)x the compute of a plain step
B = 128: 640 → compute-bound; verify step ≈ 5x a plain step at this batch
at B = 64, 4k context, plain step ≈ (141.2 + 64 × 1.34) ÷ 26.8 ≈ 8.5 ms → 7,500 tokens/s
speculation: draft 4 × ~1.5 ms + verify at the ridge ≈ 9 to 12 ms → round ≈ 15 to 18 ms for 3.36 × 64 tokens
≈ 12,000 to 14,000 tokens/s only if α holds at 0.8; at α = 0.6 it is roughly break-even
sanity: the benefit comes from converting idle compute into tokens; once the batch has used the compute, nothing is left to convert
So the decision table: batch-1 to batch-16 interactive endpoints with a good draft, use it; high-batch throughput endpoints, do not, because the same GPU-seconds serve more users through batching. The reversal condition is stated in one number: if measured α × (k+1) tokens per round does not beat the plain-step tokens the batch could have produced at the same step time, turn it off.
The reversal condition: a batch large enough that the target model is already compute-bound. Above that point the draft's verification competes with real work, the speedup goes negative, and the honest deployment makes it conditional on current batch depth rather than a global flag. Bandwidth-Bound Decode Throughput is the arithmetic that locates the crossover.
Draft choices in one line each, with the detail in the EAGLE-vs-Medusa question: a separate small model (simple, needs its own weights and KV, best when a same-family small model exists); Medusa heads (extra output heads on the target, no separate model, lower α); EAGLE-style feature-level heads (draft from the target's hidden state, higher α per parameter, the common default in vLLM and SGLang as of 2026); n-gram or prompt-lookup drafting (free, works only when output copies the prompt, as in editing tasks).
What interviewers probe next
- "Does speculation change the output?" Not with rejection sampling: accepted tokens are distributed as the target would sample them; a greedy target with a greedy draft matches token for token.
- "Why not draft 16 tokens?" Expected accepted saturates at 1 ÷ (1 - α), which is 5 at α = 0.8, and each extra draft token costs a draft step and enlarges the verify step; k of 3 to 5 is the usual optimum.
- "What does the draft's KV cost?" The draft has its own cache; for the 8B draft that is 131 KB per token, adding about 40% to the target's 328 KB per token, which reduces the sequence ceiling accordingly.
Common mistakes
- Reporting a speedup without an acceptance rate.
- Applying speculation to a batch-128 throughput endpoint and being surprised the tokens/s fell.
- Using a draft from a different tokenizer family, which needs re-tokenization at every boundary.
- Forgetting the draft's KV and weights in the memory budget.
Key takeaways
- Expected accepted tokens per round = (1 - α^(k+1)) ÷ (1 - α); 3.36 at α = 0.8, k = 4.
- Verification is nearly free only while the verify step stays memory-bound: B × (k+1) below the ridge (about 295 on H100 bf16).
- Batch-1 70B on 8 H100s goes from about 190 to about 360 tokens/s with a good draft; at batch 64 and up it breaks even or loses.
- Measure α on your traffic first; it is a property of the pair and the domain.