Explain arithmetic intensity and the roofline model. Where is the ridge point on an H100, and what does it tell you about a kernel?

TL;DR: Arithmetic intensity is FLOPs performed per byte moved from memory. The roofline says attainable throughput is min(peak FLOPS, intensity × bandwidth), and the ridge point where the two meet is peak ÷ bandwidth: for an H100 SXM, 989 TFLOPS ÷ 3.35 TB/s ≈ 295 FLOP per byte in bf16 (591 in fp8). A kernel whose intensity is below the ridge cannot reach peak no matter how well it is written; its ceiling is intensity × 3.35 TB/s, and the only ways up are to move fewer bytes or reuse each byte more.

How to approach it

Write the formula before saying anything else: attainable = min(peak, I × BW). Compute the ridge for the chip in question from the two datasheet numbers, then compute the intensity of a concrete kernel by counting its FLOPs and its bytes. Place the kernel on the roofline and say which wall it hits and what its ceiling is. Then give the two levers and the follow-up the interviewer is holding: the intensity of an operation is a property of the algorithm and the data types, and it can be changed.

A strong answer

A typical situation: a team is three weeks into optimizing a kernel's arithmetic and it has not moved. Its intensity was 2 FLOP per byte against a ridge of 295, so it was never going to move, and one division at the start would have said so.

The Roofline Model has two inputs from the hardware and one from the kernel.

From the hardware, peak compute and peak memory bandwidth. Their ratio is the ridge point, the intensity at which a kernel is exactly balanced between the two:

H100 SXM: peak dense bf16 = 989 TFLOPS, HBM3 bandwidth = 3.35 TB/s
ridge (bf16) = 989e12 FLOP/s ÷ 3.35e12 B/s = 295 FLOP per byte
ridge (fp8)  = 1,979e12 ÷ 3.35e12 = 591 FLOP per byte

other parts, same formula:
  A100 80GB:  312 ÷ 2.04 = 153      H200: 989 ÷ 4.8 = 206
  B200:      2,250 ÷ 8   = 281      MI300X: 1,307 ÷ 5.3 = 247
  TPU v6e:    918 ÷ 1.64 = 560

sanity: the ridge has crept up over generations (153 to 295) because FLOPS grew faster than
        bandwidth; a kernel that was balanced on A100 is memory-bound on H100

From the kernel, its arithmetic intensity: total FLOPs divided by total bytes moved between HBM and the chip. Count both by hand for the operation in question. Three examples every candidate should be able to do:

1. elementwise: y = x + 1 on n bf16 values
   FLOPs = n; bytes = 2n (read) + 2n (write) = 4n
   I = n ÷ 4n = 0.25 FLOP/B
   ceiling = 0.25 × 3.35e12 = 0.84 TFLOPS, 0.08% of peak; it can never be anything else

2. GEMV (decode linear layer): [1 × K] × [K × N], K = N = 8,192, bf16
   FLOPs = 2KN = 134 MFLOP; bytes ≈ 2KN (the weight matrix) = 134 MB
   I ≈ 1 FLOP/B
   ceiling = 3.35 TFLOPS (0.34% of peak); time = 134e6 ÷ 3.35e12 = 40 µs

3. GEMM: [M × K] × [K × N], M = N = K = 8,192, bf16
   FLOPs = 2MNK = 1.1e12; bytes = 2(MK + KN + MN) = 2 × 3 × 67e6 = 403 MB (each matrix once)
   I = 1.1e12 ÷ 4.03e8 ≈ 2,730 FLOP/B
   ceiling = min(989, 2,730 × 3.35) = 989 TFLOPS: compute-bound with 9x margin
   time at peak = 1.1e12 ÷ 989e12 = 1.1 ms

sanity: the three kernels span four orders of magnitude of intensity, and only the last one
        is right of the 295 ridge; this is why "the model runs at 40% MFU" is a statement about
        the mix of these three kinds of kernel, not about how well any one was written

Reading the roofline plot: the x axis is intensity on a log scale, the y axis attainable FLOPS. The sloped line is I × BW, the flat line is peak, they meet at the ridge. A kernel left of the ridge is memory-bound; its ceiling is on the slope, and measured performance below the slope means the kernel is not even saturating bandwidth (poor coalescing, low occupancy, too few bytes in flight). A kernel right of the ridge is compute-bound; below the flat roof means the arithmetic pipes are not fed (tile shapes, bank conflicts, instruction overhead, not on tensor cores).

The two levers on a memory-bound kernel follow from I = FLOPs ÷ bytes. The FLOPs are fixed by the math, so:

• Move fewer bytes for the same FLOPs. Store weights in fp8 instead of bf16: bytes halve, intensity doubles, and the fp8 tensor-core peak is also double, so the ridge in bytes-of-weights terms stays put but the kernel runs 2x faster. Fusion is the same lever: a fused layernorm plus residual plus quantize kernel reads and writes the activation once instead of three times.
• Reuse each byte more. Batching in decode: a GEMV at batch 1 reads 134 MB for 134 MFLOP; at batch 64 it reads the same 134 MB (plus a small activation) for 64 × 134 MFLOP, so intensity is 64 FLOP/B and time is barely changed while throughput is 64x. Tiling in a GEMM: each tile of A loaded into shared memory is multiplied against many tiles of B before being evicted.

A scenario that makes this concrete: a team serving a 70B model measures its decode kernels at 30 TFLOPS and asks why "utilization is 3%." The roofline answers it in one line: at batch 32 with bf16 weights the intensity is about 32 FLOP/B, the ceiling is 32 × 3.35 = 107 TFLOPS, so they are at 28% of what is possible, not 3%, and the remaining gap is the KV cache reads that the weight-only intensity ignores. The fix is a bigger batch or fp8 weights, not a better kernel.

The reversal condition: intensity above the ridge. Arithmetic Intensity by Operation places every LLM kernel on this line, and Nsight Compute's memory chart gives both inputs directly, so the classification is a lookup rather than an argument. the same team's prefill GEMMs at 4,096 tokens have intensity in the thousands and are limited by tensor-core feeding, where the levers are different.

What interviewers probe next

• "What about L2? Bytes from L2 are cheaper than from HBM." The roofline gets a second, steeper slope for L2 bandwidth; a kernel whose working set fits in the 50 MB L2 can sit left of the HBM ridge and still run fast. State which memory level the bytes come from when you compute intensity.
• "Why does the ridge for fp8 double if the bandwidth is the same?" Ridge is peak ÷ bandwidth and the fp8 peak is 1,979 TFLOPS; but fp8 weights also halve the bytes, so a decode kernel's intensity doubles at the same time. Both move; the batch at which decode becomes compute-bound stays near 295.
• "How does MFU relate to the roofline?" MFU is achieved FLOPS over peak for the whole model step. A step that is 80% compute-bound GEMMs at 70% of peak and 20% memory-bound kernels at 1% of peak lands near 40% MFU; the roofline tells you which of those two halves to work on.
• "Draw the roofline for B200 and say what changed." Peak 2,250 bf16 over 8 TB/s: ridge 281, almost the same as H100. The generation raised both roofs by about 2.3x; it did not change which kernels are memory-bound.

Common mistakes

• Computing intensity with the bytes of the output only, or forgetting that a GEMM reads two inputs and writes one.
• Quoting the ridge as a fixed number ("about 100") from a paper written for a different chip.
• Saying a kernel at 5% of peak is "badly written" without first checking whether 5% is its roofline ceiling.
• Treating the roofline as a training-only tool; it is the whole explanation of decode economics.

Key takeaways

• attainable = min(peak, I × BW); ridge = peak ÷ BW = 295 FLOP/B for H100 bf16, 591 fp8, 153 A100, 281 B200.
• Count FLOPs and bytes by hand: elementwise 0.25, GEMV 1, batched decode ≈ batch, big GEMM in the thousands.
• Left of the ridge the ceiling is I × 3.35 TB/s and the levers are fewer bytes (fp8, fusion) and more reuse (batch, tiling).
• Say which memory level the bytes come from; L2 has its own roofline.