Multi-Head Latent Attention and Sparse Indexers

TL;DR: Two independent tricks, and current models use both. Latent attention stores one compressed vector per token per layer, of width kv_lora_rank plus a small rotary part, and projects it back up into per-head keys and values inside the kernel. The cache stops scaling with head count entirely: GLM-5.3 caches 576 values per layer per token where its 64 heads at 192 dimensions would need 24,576, a factor of 42. Sparse indexing attacks the other axis. A small, cheap index scores all previous positions and keeps the top few thousand, so each query attends to a bounded set rather than to the whole prefix, which makes the attention cost per token flat in context length past that bound. A third variant, hybrid designs, replaces most layers with linear attention that keeps a fixed-size recurrent state and no growing cache at all, leaving a minority of full-attention layers. Each one changes a different term in the capacity model, and a serving engine has to support the exact combination the model uses.

Latent attention, and where the saving comes from

classic multi-head attention, cache per token per layer
  2 (K and V) x num_kv_heads x head_dim x bytes
  GLM-5.3's shape if it were plain MHA: 2 x 64 x 192 x 2 B = 49,152 B per layer

grouped-query attention, the previous generation's fix
  share KV across groups of query heads: the corpus 70B example uses 8 KV heads
  2 x 8 x 128 x 2 B = 4,096 B per layer, x 80 layers = 320 KB per token

latent attention, what the recent models do
  cache one compressed vector of width kv_lora_rank, plus the rotary part
  (kv_lora_rank + qk_rope_head_dim) x bytes
  GLM-5.3: (512 + 64) x 2 B = 1,152 B per layer, x 78 layers = 87.75 KB per token

the ratios
  MHA to latent, same model shape: 49,152 / 1,152 = 42.7x
  70B GQA to GLM-5.3 latent, per token: 320 KB / 87.75 KB = 3.6x smaller, at ten times the
    parameter count
sanity: the saving comes from the cache no longer scaling with head count, so it grows with
        model depth alone, which is why a 78-layer model caches less than an 80-layer one
        regardless of width

The cost is arithmetic. The compressed vector has to be projected back up to per-head keys and values before the attention math, every layer, every token. That work is cheap relative to the bandwidth it saves during decode, which is bandwidth-bound, and it is not free during prefill, which is compute-bound. Attention Variants: MHA, GQA, MQA and MLA covers the family.

Sparse indexing, which attacks the other axis

cost of attention per generated token, at context length L
  dense attention: proportional to L, because every position is read and scored
  sparse with top-k: proportional to L for the cheap index, plus k for the real attention
    where the index is far cheaper per position than full attention

worked, at L = 1,000,000 and k = 2,048 (GLM-5.3's index_topk)
  positions attended fully: 2,048 instead of 1,000,000, a factor of 488
  the index still touches all 1M positions, but with a much smaller per-position cost
sanity: this makes the attention term at 1M context tractable and does not make it free, and
        the KV for all 1M tokens still has to be resident, which is the 94.2 GB figure for
        GLM-5.3 at full context

The serving consequence people miss: sparse indexing reduces compute, not memory. The whole prefix stays in the cache because any position might be selected on the next token. So a model with a sparse indexer still needs the full KV budget, and the gain shows up in tokens per second rather than in gigabytes.

Hybrid designs, where most layers have no cache at all

Kimi K3 takes a third route. Published descriptions of its architecture give 93 text layers composed as 69 linear-attention layers and 24 gated latent-attention layers, interleaved roughly three to one. A linear-attention layer keeps a fixed-size recurrent state rather than a cache that grows with context.

KV growth in a hybrid stack
  layers with a growing cache: 24 of 93, about 26%
  layers with fixed state:     69 of 93, contributing a constant regardless of context
  so KV per token is roughly a quarter of what an all-latent stack of the same depth costs

what this demands of the engine
  a single cache manager that handles two kinds of state at once: paged blocks for the
    full-attention layers and fixed recurrent state for the linear ones
  the vLLM project's launch note for Kimi K3 describes exactly this, and notes that prefix
    caching was off by default while the hybrid cache design settled
sanity: an architecture that saves memory can still be unservable on day one if no engine
        implements its cache manager, which is why day-zero support is a real procurement
        criterion rather than a nicety

Which one changes which number

What interviewers are listening for

That you separate memory from compute. Candidates routinely say "sparse attention makes long context cheap" without noticing that it makes the compute cheap and leaves the memory exactly where it was. The strong answer names which technique moves which term, computes one of them, and adds the engine-support point, because a model whose attention no engine implements is not deployable no matter how good the design is. Interviewers also probe the prefill side, since latent attention adds arithmetic there while saving bandwidth during decode.

Key takeaways

• Latent attention caches kv_lora_rank + qk_rope_head_dim per layer per token: 576 values for GLM-5.3 against 24,576 under plain multi-head attention, a factor of 42.7.
• The saving stops scaling with head count, so KV grows with depth alone.
• Sparse indexing bounds attended positions at index_topk, 2,048 for GLM-5.3 and 512 for DeepSeek V4-Flash, cutting compute and not memory.
• Hybrid stacks such as Kimi K3's 69 linear plus 24 full-attention layers cut KV to roughly a quarter and require an engine with a mixed cache manager.
• Day-zero engine support for the exact attention and quantization combination is a deployment requirement, not a detail.