How big is the KV cache for Llama 3.1 70B at a 128k context?

TL;DR: KV bytes per token = 2 × layers × KV heads × head dim × bytes = 2 × 80 × 8 × 128 × 2 = 320 KB in bf16. At 131,072 tokens that is 320 KB × 131,072 ≈ 42 GB for one sequence, about a third of the 141 GB of weights, and it halves to 21 GB with an fp8 cache.

How to approach it

Ask which precision the cache is stored in, since fp8 KV is common and halves the answer. Then say the formula with every variable named before substituting, and say "two, for K and V" as the first factor so it is never dropped. Pull the four architecture numbers (80 layers, 8 KV heads, 128 head dimension, 2 bytes) and compute the per-token figure first, then multiply by context. Finish by placing the result against a card: "42 GB is more than half an H100, for one user."

A strong answer

A typical situation: a capacity plan uses the 64 attention heads instead of the 8 KV heads, comes out 8x too pessimistic, and concludes long context is unservable. The two fields sit next to each other in the config file.

Every token in a live sequence keeps one key and one value vector per KV head at every layer, so that later tokens can attend to it without recomputation. The size therefore depends on the attention shape, never on the parameter count. Llama 3.1 70B uses grouped-query attention: 64 query heads share 8 KV heads, and the cache stores only the 8.

inputs (from config.json):
  n_layers   = 80
  n_kv_heads = 8        (num_key_value_heads; the 64 is num_attention_heads, do not use it)
  d_head     = 128      (hidden 8192 ÷ 64 query heads)
  bytes      = 2        (bf16)

KV per token = 2 × n_layers × n_kv_heads × d_head × bytes
             = 2 × 80 × 8 × 128 × 2
             = 327,680 B = 320 KiB exactly (327.68 KB decimal; say KiB and mean it)

per sequence at 128k:
  context    = 131,072 tokens
  KV         = 327,680 × 131,072 = 4.29e10 B ≈ 42.9 GB

sanity: 42 GB is 30% of the 141 GB bf16 weights, and more than half of one 80 GB H100,
        so a single full-context user costs the fleet as much memory as a third of the model.

The number is usually needed in a capacity context, so carry it one step further. On an 8 × H100 node serving bf16 weights there is roughly 435 GB of usable memory after weights and a 10% reserve; at 42.9 GB per full-context sequence that is 10 concurrent 128k users. With an fp8 cache (bytes = 1) each sequence drops to 21.5 GB and the count doubles to 20. Move the weights to fp8 as well and the free budget rises to about 505 GB, or 23 users at 128k.

users at 128k on 8 × H100, bf16 weights:
  free = 8 × 80 × 0.9 − 141 = 435 GB
  bf16 KV: 435 ÷ 42.9 ≈ 10 sequences
  fp8 KV:  435 ÷ 21.5 ≈ 20 sequences

The MoE comparison is worth one sentence because it shows why the formula matters: Mixtral 8x7B has 47B parameters and the same 32-layer, 8-head, 128-dim shape as Llama 8B, so its cache is 128 KB per token, less than half the 70B's, despite the parameter count. DeepSeek-V3 with multi-head latent attention stores 61 × (512 + 64) × 2 ≈ 70 KB per token, a fifth of the 70B's, with 671B parameters. The cache is architecture arithmetic.

What reverses the "42 GB" answer is a different serving precision or a different model, never the batch size; the per-sequence cost is fixed, and batch only multiplies it. The KV cache sizing page carries the derivation for four models.

The reversal condition: an MLA model, where the formula itself changes. DeepSeek-V3 stores a 576-wide latent per layer instead of full keys and values, so its cache is about 70 KB per token despite 671B parameters, and applying the GQA formula to it overestimates by more than three times. KV Cache Sizing works both shapes, and Attention Variants: MHA, GQA, MQA and MLA is where the layout difference comes from. Setting --kv-cache-dtype fp8 halves whichever number you land on, which is the cheapest lever here.

What interviewers probe next

• "How would you cut it?" In order of effect: fp8 KV (halves it, no architecture change), paged allocation so reserved-but-unused context is not charged, prefix caching so a shared system prompt is stored once, and, at model selection time, fewer KV heads or MLA.
• "What is it at 8k?" 320 KB × 8,192 = 2.6 GB. A batch of 64 such sequences is 168 GB, which is why short-context batches on the 70B still need a multi-card node.
• "Does a longer context change the per-token cost?" No; the per-token figure is constant, and the sequence cost grows linearly. What grows faster is attention compute during prefill, which is a separate question.
• "Why does GQA help here?" It divides the cache by the query-to-KV head ratio, 8 on this model, while keeping the query heads for quality.

Common mistakes

• Using 64 heads instead of 8: 2 × 80 × 64 × 128 × 2 = 2.6 MB per token, 343 GB per 128k sequence, and a capacity plan eight times too pessimistic.
• Dropping the factor of 2 and reporting 160 KB per token; the fix is to say "K and V" before anything else.
• Multiplying by the parameter count somewhere, usually because the candidate remembers that "bigger models have bigger caches" and reaches for the wrong big number.
• Reporting 40 GiB against a card quoted in GB, so the fit looks better than it is.

Key takeaways

• KV per token = 2 × layers × KV heads × head dim × bytes; Llama 3.1 70B in bf16 is 320 KB.
• 128k context: 42 GB per sequence in bf16, 21 GB in fp8. 8k context: 2.6 GB.
• The KV-head count is 8, not 64; that one input is where the factor of eight hides.
• On 8 × H100 with bf16 weights, about 10 full-context users fit; fp8 KV doubles it.