TL;DR: Cache per token is 2 × layers × KV heads × head dim × bytes for MHA and GQA, so a 70B-class model with 64 heads stores 2.6 MB per token and Llama 3.1 70B with 8 KV heads stores 328 KB, an 8x cut for a small quality cost. MLA stores one compressed latent plus a small RoPE key per layer with no factor of two: DeepSeek-V3's 61 × (512 + 64) × 2 B = 70 KB per token, which is why a 671B model holds more context per GPU than a dense 70B.
How to approach it
Write the general formula first and name the variable that changes between the three: KV heads. Ask which models the interviewer wants compared, or offer Llama 3.1 70B (GQA), a hypothetical 64-KV-head version of it (MHA) and DeepSeek-V3 (MLA). Compute each per-token figure, scale one of them to a full sequence so the difference is visible in gigabytes, then say what each layout trades away.
A strong answer
A typical situation: two models of the same parameter count are benchmarked on the same card and one holds four times the concurrent users. The difference is in config.json, in a field most capacity plans never read.
The KV cache stores, for each layer and each token, whatever the attention mechanism needs to reconstruct keys and values. Attention Variants: MHA, GQA, MQA and MLA differ in how much of that is kept per head.
general (MHA, GQA, MQA): KV per token = 2 × layers × KV heads × head dim × bytes
MHA, 70B-class, 64 KV heads (every query head has its own K and V):
= 2 × 80 × 64 × 128 × 2 = 2,621,440 B ≈ 2.6 MB per token
GQA, Llama 3.1 70B, 8 KV heads (8 query heads share one KV head):
= 2 × 80 × 8 × 128 × 2 = 327,680 B ≈ 328 KB per token (8x smaller)
MQA, 1 KV head:
= 2 × 80 × 1 × 128 × 2 = 40,960 B ≈ 41 KB per token (64x smaller than MHA)
MLA, DeepSeek-V3: KV per token = layers × (latent + RoPE key) × bytes, no factor of 2
= 61 × (512 + 64) × 2 = 70,272 B ≈ 70 KB per token
sanity: at 32k context one sequence costs 86 GB (MHA), 10.7 GB (GQA-8), 1.3 GB (MQA), 2.3 GB (MLA-V3)
The 32k row is where it lands. On a single 80 GB H100 with weights elsewhere, an MHA 70B cannot hold even one 32k sequence's cache; GQA-8 holds 7; DeepSeek-V3's MLA holds about 34, and that is a 671B model.
Why the layouts differ. MHA keeps a distinct K and V projection per query head, which gives the model the most expressive attention and the largest cache. GQA groups query heads (8 per group in Llama 3) onto a shared K and V head, so the cache shrinks by the group size while the projection weights shrink by the same ratio; Llama 3 reports the quality cost as small, and every major dense model since 2023 has used it. MQA takes the group size to the whole layer; it is the cheapest and it costs noticeable quality on large models, so it is mostly seen in older or small models.
MLA is a different idea. Instead of storing K and V, it stores a low-rank latent c (512 wide in V3) from which K and V for all heads are reconstructed by up-projection matrices at attention time, plus a small decoupled RoPE key (64 wide) because rotary position embedding does not commute with the compression. The factor of 2 disappears because one latent serves both K and V. The cache shrinks to 70 KB per token for 128 heads of attention, and DeepSeek reports quality at or above MHA because the latent is trained, not a post-hoc compression.
What MLA costs: the up-projection is a matmul at every decode step per token per layer, so the attention kernel does more compute per byte read. That is a good trade for decode, which is bandwidth-bound and has compute to spare. The kernels are custom (FlashMLA and the SGLang and vLLM MLA paths), and the "absorbed" form that folds the up-projection into the query projection is what makes it fast; a naive implementation reconstructs full K and V and loses the memory advantage inside the kernel.
| Layout | KV per token (70B-class or V3) | 32k sequence | Quality | Kernel |
|---|---|---|---|---|
| MHA, 64 heads | 2.6 MB | 86 GB | reference | standard |
| GQA, 8 heads | 328 KB | 10.7 GB | near reference | standard |
| MQA, 1 head | 41 KB | 1.3 GB | measurable loss at scale | standard |
| MLA (V3 dims) | 70 KB | 2.3 GB | at or above reference | custom, absorbed form |
KV Cache Sizing is the arithmetic this feeds. The reversal condition: a short-context product, where the cache never grows enough for the layout to matter and the weights dominate memory regardless. --kv-cache-dtype fp8 is a bigger lever than the attention variant at 2k context, and a smaller one at 128k. The decision for a serving engineer is rarely which layout to use (the model decides); it is what the layout permits. GQA-8 at 70B is the case the rest of this bank sizes. MLA is why serving DeepSeek-V3 behaves differently from a dense model of similar active size: memory like 671B for weights, cache like a 20B model.
What interviewers probe next
- "Can you convert an MHA checkpoint to GQA?" Yes, by mean-pooling the K and V heads in each group and fine-tuning briefly (the GQA paper's recipe); quality recovers with a small fraction of pretraining tokens.
- "Why does MLA need a separate RoPE key?" Rotary embeddings multiply keys by a position-dependent rotation; applying that inside the compressed latent would break the up-projection trick, so a small rotated key is stored uncompressed alongside the latent.
- "How does fp8 KV combine with these?" It halves any of them: GQA-8 drops to 164 KB per token and MLA to 35 KB; the accuracy effect is small for K and V but should be measured per model.
Common mistakes
- Using attention heads instead of KV heads for GQA and getting the MHA number.
- Applying the factor of 2 to MLA.
- Saying MLA is "just a smaller GQA" without the up-projection cost.
- Treating MQA as free; it is where quality loss is measurable.
Key takeaways
- MHA/GQA/MQA: 2 × layers × KV heads × head dim × bytes; the KV head count is the lever.
- 70B-class: 2.6 MB (64 heads), 328 KB (8 heads), 41 KB (1 head) per token in bf16.
- MLA: layers × (latent + RoPE) × bytes; DeepSeek-V3 is 70 KB per token, no factor of 2.
- MLA moves cost from bytes to compute in the attention kernel, which suits bandwidth-bound decode.