AI Infra Interviews logo
← GPU & Accelerator Architecture / 11
mediumNewNVIDIAFireworks

A kernel runs at 25% occupancy. Is that a problem? Walk me through what occupancy buys and when you would leave it low on purpose.

Occupancy is a means, and the end is enough bytes in flight to cover memory latency. Little's law gives the number of loads an SM needs outstanding; the register file gives the warps you can afford; the kernels that win on H100 usually run at 25% to 50% occupancy with big tiles and no spills.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

Occupancy is a means, and the end is enough bytes in flight to cover memory latency. Little's law gives the number of loads an SM needs outstanding; the register file gives the warps you can afford; the kernels that win on H100 usually run at 25% to 50% occupancy with big tiles and no spills.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced
⚡ Kernels & Compilers🔒 Premium
Occupancy and Register PressureOccupancy is the fraction of an SM's 64 warp slots that are resident, and it is capped by the 65,536 registers and 228 KB of shared memory each block consumes. It decides how much memory latency the hardware can hide for free, but the fastest kernels on a GPU routinely run at 25 percent, so the interview skill is knowing when to raise it and when to stop.
Foundational
🧩 GPU & Accelerator Architecture
GPU Memory HierarchyA GPU has four places a byte can live, and they differ by a thousandfold in bandwidth: registers, shared memory on the SM, a chip-wide L2, and HBM off-chip. Almost every kernel optimization is a decision about which level a value is read from and how many times. Knowing the sizes and bandwidths for an H100 cold is what lets you say why a kernel is slow before you profile it.
Foundational
🧩 GPU & Accelerator Architecture
GPU Execution ModelA GPU hides memory latency with parallelism instead of caches: thousands of threads in flight, scheduled in warps of 32, pinned to streaming multiprocessors that switch between warps for free whenever one stalls. Every performance conversation in an AI infra loop, from occupancy to why decode is slow, rests on this one mechanism.
Advanced
⚡ Kernels & Compilers🔒 Premium
Tiled Matrix MultiplicationA matrix multiply has enough reuse to be compute-bound, but only if the kernel captures that reuse in shared memory and registers instead of re-reading HBM. Tiling is how: a block owns an output tile, streams K-slices of A and B through shared memory, and each thread accumulates a small register tile. It is the live-coding exercise that separates people who know the roofline from people who have climbed it.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackExplain what HBM is and why memory bandwidth, not compute, is the wall for LLM inference.Next in this trackWhat actually changes with Blackwell and the NVL72 rack, and what does it do to how you would serve a large MoE model?Next in this trackYou have trained on GPUs. What transfers to training on TPUs, and what do you have to relearn?
FEDITOR'S NOTE

Scored on the register arithmetic that produces the occupancy number, a Little's-law estimate of how much latency hiding is actually required, and the judgment that spilling registers to raise occupancy is usually a loss.

DISCUSSION · 0

No comments yet — be the first to share your approach.