Prefill vs Decode

TL;DR: Prefill processes every prompt token in one pass, so it is a large matrix multiply that saturates tensor cores; decode processes one token per pass, so it re-reads all the weights and the whole KV cache to produce a few FLOPs per byte and sits far below the roofline ridge. Prefill sets time to first token, decode sets time per output token, and they want different batch sizes and different hardware.

Two phases, one request

A chat request arrives as a prompt of a few hundred to a few thousand tokens. The engine runs the entire prompt through the model in a single forward pass, writing the key and value vectors of every prompt token into the KV cache and producing exactly one output token. That is prefill. From then on the model runs one forward pass per new token, reading the cache it wrote earlier and appending one more entry each step. That is decode, and it repeats until the model emits a stop token or hits the length limit.

The two phases do the same math per token: roughly 2 FLOPs per parameter. What differs is how many tokens share each read of the weights. In prefill, 2,048 tokens share one pass over the weights, so the arithmetic intensity is thousands of FLOPs per byte. In decode, at batch size 1, a single token shares that pass, and intensity collapses to about 1 FLOP per byte.

The numbers on an H100

Take Llama 3.1 70B in bf16 (70.6B parameters, about 141 GB of weights) on an H100 SXM: 989 dense bf16 TFLOPS and 3.35 TB/s of HBM bandwidth, so the ridge point is about 295 FLOP/byte.

Prefill of a 2,048-token prompt is 2 × 70.6e9 × 2,048, about 289 TFLOP. At peak that is 292 ms on one card, and at a realistic 60 to 70 percent of peak closer to 450 ms. Spread across four cards with tensor parallelism it lands around 120 ms. The point: prefill time scales with prompt length and with FLOPS, and the KV cache barely enters into it.

A decode step at batch 1 must stream all 141 GB of weights from HBM. At 3.35 TB/s that is about 42 ms, or roughly 24 tokens per second, and the tensor cores are idle over 99 percent of the time because the step performs 141 GFLOP where the card could do 989 TFLOPS in that window. Doubling the batch to 2 nearly doubles throughput at almost no latency cost, and this keeps working until the batch is large enough to cross the ridge or until the KV reads start to dominate the weight reads.

What each phase wants

The H200 row is the one to remember. It has the same compute die as the H100 and about 43 percent more bandwidth, so it speeds up decode and does nothing for prefill. An interviewer who asks "which phase does the upgrade help" is checking whether you know this.

Where it breaks in production

A mixed fleet runs both phases on the same GPUs, and that is where the trouble starts. A 30k-token prompt arriving at a replica pauses every in-flight decode stream for the entire prefill, so the streaming users see a stall of several hundred milliseconds. That is head-of-line blocking, and the standard fixes are chunked prefill (cap the prefill tokens per scheduler step) or disaggregation (separate pools with a KV transfer between them).

The second trap is reading nvidia-smi. Utilization reports 100 percent during decode because a kernel is always resident, while the tensor cores are mostly waiting on HBM. The number that tells the truth is the ratio of dram__throughput to sm__throughput in Nsight Compute, or DCGM's DCGM_FI_PROF_PIPE_TENSOR_ACTIVE next to DCGM_FI_PROF_DRAM_ACTIVE. Decode shows high DRAM activity and low tensor activity; prefill shows the reverse.

What interviewers are listening for

The screening question is "is decode compute-bound or memory-bound, and why". The strong answer states the intensity: 2 FLOPs per parameter per token against 2 bytes per parameter per step, so about 1 FLOP per byte at batch 1, against a ridge near 300 on an H100. The follow-up they hold back is "so why not batch decode until it is compute-bound". The answer that fails says "yes, batch to 300". The answer that lands notes that each sequence also re-reads its own KV cache every step, 320 KB per token for this model, so at 8k context a batch of 64 reads about 170 GB of cache per step on top of the 141 GB of weights, and the batch hits a memory ceiling before it reaches the ridge.

Common misconceptions

Candidates say prefill is "the slow part". Per token it is the fast part by two orders of magnitude; it only feels slow because it happens all at once before the first byte streams.

Candidates say a bigger GPU fixes latency. More FLOPS shorten TTFT and do nothing for TPOT; more bandwidth does the reverse. Ask which metric is breached before choosing the card.

Candidates say decode at batch 1 uses the GPU fully because utilization shows 100 percent. It uses the memory system fully and the compute almost not at all.

Key takeaways

• Prefill is one compute-bound pass over the prompt and sets TTFT; decode is one memory-bound pass per token and sets TPOT.
• On an H100, Llama 3.1 70B decodes at about 24 tokens per second single-stream because 141 GB of weights cross 3.35 TB/s of HBM each step.
• Batching decode is nearly free throughput until KV reads, not the ridge, become the ceiling.
• Compute upgrades help prefill; bandwidth upgrades (H200 over H100) help decode.
• Mixed-phase replicas suffer head-of-line blocking; chunked prefill and disaggregation are the two remedies.