TL;DR: Attention at each step needs the keys and values of every earlier token in every layer, and recomputing them would cost a full prefill per generated token, so the engine stores them. For Llama 3.1 70B in bf16 that is 2 × 80 layers × 8 KV heads × 128 dims × 2 bytes = 327,680 bytes per token, about 1.3 GB per 4k-token sequence, and it grows linearly with context and with the number of concurrent sequences until it, not the weights, decides how many users fit.
How to approach it
Start from why the cache exists: the attention score for the new token is a dot product against every previous key, and the output is a weighted sum of every previous value, so those tensors must be available at every step. Ask which model, because the per-token cost depends on layers, KV heads and head dimension, and say you will pick Llama 3.1 70B if they do not care. Write the per-token formula first, then multiply by context, then by batch, and finish by comparing the total against the memory left after weights. That order makes the growth argument fall out of the arithmetic instead of being asserted.
A strong answer
A typical situation: a replica serves 40 users comfortably and falls over at 48, with the GPUs at 30% utilization the whole time. Memory ran out, not compute, and the growth is linear in a number nobody was watching.
Generation is autoregressive. Token t attends over tokens 1 to t-1, and token t+1 attends over 1 to t. Without a cache the engine would rerun every layer's key and value projections over the whole prefix at every step, which turns an O(n) generation into O(n²) compute. The KV Cache trades memory for that compute: each layer's K and V for each token are written once, during prefill for the prompt and during each decode step for generated tokens, and read at every later step.
The size follows from the architecture. Only KV heads count, not attention heads, because grouped-query attention shares one K and V head across several query heads:
inputs (Llama 3.1 70B, bf16)
layers = 80, KV heads = 8, head dim = 128, bytes per element = 2
KV per token = 2 (K and V) × layers × KV heads × head dim × bytes
= 2 × 80 × 8 × 128 × 2
= 327,680 B ≈ 328 KB
per sequence at 4,096 tokens: 328 KB × 4,096 ≈ 1.34 GB
per sequence at 32,768 tokens: 328 KB × 32,768 ≈ 10.7 GB
per sequence at 131,072 tokens (the published max): ≈ 43 GB
sanity: one 128k sequence does not fit beside the weights on a single 80 GB card,
and 8 users at 32k need 86 GB of cache on top of 141 GB of weights
Growth comes from two multipliers. Context: every generated token appends one more 328 KB slice, so a sequence that starts at 1,000 tokens and writes 2,000 more triples its footprint while it runs. Concurrency: each active sequence has its own cache, so the pool the scheduler must manage is per-token cost × context × running sequences. Against a fixed budget, that product is the admission limit:
8 × H100 SXM, 640 GB total, 90% usable = 576 GB
weights bf16 = 70.6e9 × 2 B = 141.2 GB
KV budget = 576 - 141.2 ≈ 435 GB
concurrent sequences at 4k = 435 GB ÷ 1.34 GB ≈ 324
concurrent sequences at 32k = 435 GB ÷ 10.7 GB ≈ 40
sanity: the same fleet holds 8x fewer users when prompts are 8x longer; compute did not change
That is the reason a serving team watches KV utilization rather than SM utilization. When the pool is full, the engine either refuses admission, preempts a running sequence and recomputes it later, or swaps blocks to host memory. All three show up as latency, and none of them show up as a busy GPU.
The reversal condition: short fixed-length outputs, where the cache never grows enough to bind and the constraint returns to compute. That is rare in a chat product and common in a scoring pipeline, and it changes which lever below is worth pulling. --kv-cache-dtype fp8 is the cheapest of them.
Three levers shrink the cache. Fewer KV heads (GQA with 8 heads is already 8x smaller than the 64-head MHA equivalent, and MLA compresses further; see Attention Variants: MHA, GQA, MQA and MLA). Fewer bytes per element (fp8 KV halves it to 164 KB per token). Fewer stored tokens (prefix sharing, sliding-window layers). The engine-side lever is PagedAttention, which does not shrink the cache but stops it from being wasted on reserved-but-unused space.
What interviewers probe next
- "Why does GQA reduce the cache but not the weights much?" KV projection weights are a small slice of each layer; the cache scales with KV heads directly, so cutting 64 heads to 8 cuts cache 8x while trimming parameters by a few percent.
- "What is the KV cache for an 8B model?" 2 × 32 × 8 × 128 × 2 = 131 KB per token, so a 4k sequence is 537 MB and one H100 holds roughly 400 of them beside 16 GB of weights.
- "Does the cache affect speed or only capacity?" Both: every decode step reads the whole cache of every running sequence, so at 64 sequences × 4k the step reads 86 GB of KV on top of 141 GB of weights and slows by about 60%.
Common mistakes
- Counting attention heads (64) instead of KV heads (8) and getting 2.6 MB per token.
- Forgetting the factor of 2 for K and V.
- Quoting the cache in "per request" terms without a context length, which makes the number meaningless.
- Treating the cache as static after prefill when it grows by one slice per generated token per sequence.
Key takeaways
- KV per token = 2 × layers × KV heads × head dim × bytes; 328 KB for Llama 3.1 70B in bf16, 131 KB for 8B.
- Per sequence = per token × context: 1.34 GB at 4k, 10.7 GB at 32k, 43 GB at 128k for 70B.
- Concurrency limit = (usable memory minus weights) ÷ per-sequence cache; 8 H100s hold about 324 users at 4k and 40 at 32k.
- The cache is read every decode step, so it costs bandwidth as well as capacity.