The KV for one sequence is scattered across fixed-size pages, so the kernel reads a table of physical block numbers before it can read any keys. What the indirection costs, what the table costs in memory, why decode must split the KV dimension across thread blocks, and how the partial softmax states combine.
Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.
The KV for one sequence is scattered across fixed-size pages, so the kernel reads a table of physical block numbers before it can read any keys. What the indirection costs, what the table costs in memory, why decode must split the KV dimension across thread blocks, and how the partial softmax states combine.
Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.
Scored on the block-table indirection inside the inner loop, on knowing why decode needs a split across KV blocks to fill the GPU, and on the memory arithmetic that shows the indirection is nearly free.
No comments yet — be the first to share your approach.
