Describe the GPU memory hierarchy. Where can a byte live on an H100, and what does each level cost?

TL;DR: On an H100 a byte can live in a register (33 MB of them across the chip, tens of TB/s, one cycle), in shared memory or L1 (228 KB per SM, register-like bandwidth, about 30 cycles), in the 50 MB L2 (several TB/s, about 200 cycles), in 80 GB of HBM3 (3.35 TB/s, about 600 ns), or in host memory across PCIe (64 GB/s per direction, microseconds). Each step down is roughly 10x bigger and several times slower, and every GPU performance question reduces to which level the hot loop is actually reading from.

How to approach it

List the levels from the SM outward, giving a size and a bandwidth for each so the interviewer hears numbers rather than names. Say who controls each level: the compiler for registers, the programmer for shared memory, the hardware for L1 and L2. Then show that the hierarchy is a diagnostic tool by working one example where moving a working set up one level changes the time. Ask whether they care about the host side too, since PCIe and pinned memory are where data loading questions live.

A strong answer

A typical situation: a kernel is 4x slower than the FLOP count says it should be, and the team spends a week on the arithmetic. The bytes were coming from HBM every iteration when the working set would have fitted in L2, and the fix was a tiling change rather than a math change.

The GPU Memory Hierarchy on an H100 SXM, from the arithmetic units outward:

The register figure surprises people, so derive it:

registers per SM = 65,536 × 4 B = 262,144 B = 256 KB
per chip         = 256 KB × 132 SMs ≈ 33.8 MB
sanity: that is more on-chip storage than the L2 (50 MB) is often credited with by people who
        forget registers exist; it is why the GPU can hold thousands of threads' state at once

The aggregate register and shared-memory bandwidths are not vendor-published per se, but the shape is what matters: each SM can feed its tensor cores from shared memory at a rate that is dozens of times the SM's share of HBM. Per SM, HBM delivers only 3.35 TB/s ÷ 132 ≈ 25 GB/s. That gap between "what an SM can consume" and "what HBM can deliver to it" is the whole reason tiling exists.

Here is the hierarchy used as a diagnostic, on the workload every AI infra loop asks about. Multiplying a 4,096 × 4,096 bf16 matrix by a 4,096-wide vector (a decode-time GEMV):

weights = 4,096 × 4,096 × 2 B = 33.5 MB
reads from HBM once:  33.5 MB ÷ 3.35 TB/s = 10 µs
FLOPs = 2 × 4,096 × 4,096 = 33.5 MFLOP; at 989 TFLOPS that is 0.03 µs
sanity: the arithmetic is 300x faster than the read, so this kernel is a memory copy with a
        multiply attached; the only question is which level the 33.5 MB comes from
if the same matrix is used again by the next kernel: 33.5 MB < 50 MB L2, so a second pass can
        come from L2 at several TB/s instead of HBM, if nothing evicted it in between

That last line is how the hierarchy earns its keep in an interview. The same kernel is fast or slow depending on whether its working set fits the level above the one you assumed. Three rules follow:

• Registers are the fastest memory and the scarcest. A kernel that needs 128 registers per thread halves the resident warps compared with one that needs 64. When the compiler runs out, it spills to "local memory," which is a name for HBM with an L1 in front; spills show in nvcc -Xptxas -v as spill stores and spill loads.
• Shared memory is the programmer's explicit staging area. Tiled GEMM and FlashAttention exist to load a tile from HBM once into shared memory, then reuse it dozens of times from there. It has 32 banks of 4 bytes; two threads in a warp hitting different addresses in the same bank serialize, which is the bank conflict problem.
• L2 is shared by every SM and is not under your control, mostly. 50 MB sounds like a lot until you note that a single layer of a 70B model is about 1.6 GB in bf16. L2 helps when consecutive kernels touch the same tens of megabytes, or when many blocks of one kernel read the same rows.

Host memory deserves one sentence in this answer: pageable host memory cannot be DMA'd, so a cudaMemcpy from it goes through a pinned staging buffer and runs at a fraction of PCIe speed; cudaHostAlloc (pinned) memory runs at the full 64 GB/s per direction. That is the difference between a data loader that keeps up and one that does not.

The decision this hierarchy drives is always the same: find the level the hot loop is reading from, compute the time at that level's bandwidth, and either move the working set up a level (tiling, fusion, residency) or accept that level's speed and stop optimizing arithmetic that is not the bottleneck. The reversal condition: the kernel is compute-bound, which on an H100 means arithmetic intensity above roughly 295 FLOP per byte from HBM. Above that line the level a byte lives at stops mattering and the Roofline Model puts you on the flat roof, where the only remaining lever is the arithmetic itself. Nsight Compute's memory chart names the level in one screen and settles which side of the line you are on.

What interviewers probe next

• "Why is shared memory faster than L1 if they are the same SRAM?" They are the same physical array, split by configuration; shared memory is faster in practice because the programmer guarantees the hit, so there are no tag checks or misses.
• "What is the L2 bandwidth, roughly?" Several times HBM; enough that a kernel whose working set fits in L2 stops being HBM-bound and starts being bound by L2 or by instruction issue. Measure it with a microbenchmark before quoting a number.
• "What changes on MI300X?" Same shape with different sizes: 192 GB of HBM3 at 5.3 TB/s, a 256 MB Infinity Cache between L2 and HBM, and 64 KB local data share per compute unit instead of 228 KB.
• "How big is the KV cache relative to these?" 320 KB per token for Llama 3.1 70B in bf16, so a 4k-token sequence is 1.3 GB: far past L2, and the reason decode reads HBM every step.

Common mistakes

• Reciting the levels without a single number, or with the wrong order of magnitude (calling HBM "fast" without saying compared with what).
• Treating L1 and L2 like CPU caches that make a thread fast. They save bandwidth on reuse; latency is covered by other warps.
• Forgetting registers are memory, and then being unable to explain why a kernel with 200 registers per thread runs at 12% occupancy.
• Reading data from pageable host memory in a loop and blaming PCIe.

Key takeaways

• Six places a byte lives: registers (256 KB per SM), shared/L1 (228 KB per SM), L2 (50 MB), HBM (80 GB at 3.35 TB/s), host over PCIe (64 GB/s per direction), peer over NVLink (900 GB/s).
• Per SM, HBM is only about 25 GB/s; everything about tiling exists to close that gap with on-chip reuse.
• Diagnose by asking which level the hot loop reads and computing time at that level's bandwidth.
• Registers are the binding resource for occupancy; spills are HBM traffic in disguise.