For bf16 weights the decode intensity equals the batch, so the ridge is at batch 295 on an H100. Then the KV cache enters and the answer changes shape: the projections go compute-bound at 300 while the attention reads never do, and at 8k context the step as a whole stays memory-bound at any batch.
Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.
For bf16 weights the decode intensity equals the batch, so the ridge is at batch 295 on an H100. Then the KV cache enters and the answer changes shape: the projections go compute-bound at 300 while the attention reads never do, and at 8k context the step as a whole stays memory-bound at any batch.
Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.
The interviewer wants the weights-only derivation (intensity = batch, ridge = 295) and then wants the candidate to add the KV term and notice that it caps the intensity. A candidate who stops at 300 has the textbook answer; one who shows the cap has run a server.
No comments yet — be the first to share your approach.
