Practice tests · 30 questions

LLM Inference & Serving: the practice test

Prefill versus decode, the KV cache, PagedAttention and continuous batching, chunked prefill, speculative decoding, disaggregated serving, quantization, vLLM, SGLang and TensorRT-LLM, multi-LoRA and routing: hosting open-weight models at a latency SLO and a cost you can defend. This test drills exactly that: 11 easy, 11 medium and 8 hard questions, every one explained, every explanation linking into the worked material.

Set up your test

Topic

How confident are you feeling?

Questions

11 in this pool · about 7 min

Reveal answers

Sample questions, answered

easy · sample

Why do prefill and decode behave so differently on the same GPU, and what does that imply for hardware?

Prefill runs on the CPU while decode runs on the GPU, so the two phases need different host configurations into an embedding table that fits in the L2 cache

✓Prefill is compute-bound over the whole prompt; decode streams every weight per token and is bandwidth-bound

Decode is compute-bound because sampling is expensive; prefill is a cheap lookup

They are identical in cost; the difference is only in how the engine logs them

Prefill multiplies the whole prompt through the weights in one pass, so its arithmetic intensity is high (thousands of FLOPs per byte) and it saturates the tensor cores. Decode generates one token per step and must stream every weight from HBM for about 2 FLOPs per weight, an intensity near 1 FLOP per byte, so memory bandwidth sets its speed. That split is why disaggregated serving puts prefill on compute-rich parts and decode on bandwidth-rich ones, and why batching helps decode far more than prefill.

easy · sample

What does the KV cache store, and why does it grow during a response?

The model weights for the layers already computed during this request, so they do not have to be reloaded from HBM

The tokenizer vocabulary, so it expands as new words appear

✓The keys and values of every past token at every layer, one entry per token per layer

The sampling logits for every past step, used to enforce repetition penalties

Attention for a new token needs the keys and values of every previous token at every layer. Computing them once and caching them turns a quadratic recomputation into a linear read, at the cost of memory that grows by one entry per layer for every token in the context (prompt plus generated). For a 70B model in bf16 that is about 320 KB per token, so a 32k-token sequence holds about 10.5 GB, which is why the cache, not the weights, is what limits concurrency.

Go deeper than the quiz

A practice test measures recall. The material it draws from teaches the reasoning: