TL;DR: Elementwise and reduction operations are bandwidth-bound: their cost is the bytes they move, not the arithmetic. A chain like
y = dropout(gelu(x + b))as three kernels reads and writes the activation three times; fused into one kernel it reads once and writes once, which is a third of the traffic and a third of the time. At small tensor sizes there is a second saving: each kernel launch costs a few microseconds of CPU and GPU overhead, and a decode step that launches 400 small kernels spends more time launching than computing. Fusion has three common shapes: elementwise chains, a reduction with its consumers (softmax as one kernel, layernorm as one kernel), and a GEMM with its epilogue (bias, activation, residual add written in the GEMM's output stage). What fusion cannot do is make a compute-bound operation faster: a large GEMM at 70% of peak FLOPS is limited by the tensor cores, and fusing its epilogue saves the epilogue's traffic, not the GEMM's time.
How to approach it
Do the byte arithmetic for a concrete chain, unfused and fused. Add launch overhead and say when it dominates. Then the three fusion shapes with an example each, and how the tools do it (torch.compile, Triton, hand-written). Close with the limits: compute-bound ops, fusions that break parallelism, and register pressure.
A strong answer
A typical situation: a transformer MLP block in eager PyTorch runs the GEMM, then a bias add, then GELU, then dropout as four kernels; the profiler shows the GEMM at 60% of the block's time and the three elementwise kernels at 40% despite doing almost no arithmetic.
The arithmetic:
activation: batch 32 × seq 4,096 × hidden 16,384 (the MLP's intermediate) in bf16 = 32 × 4,096 × 16,384 × 2 B = 4.3 GB
unfused: bias add reads 4.3 + writes 4.3; GELU reads 4.3 + writes 4.3; dropout reads 4.3 + writes 4.3 (+ mask)
→ ~26 GB moved; at 3 TB/s ≈ 8.6 ms
fused elementwise: read 4.3, write 4.3 → 8.6 GB → 2.9 ms; 3× faster, and the arithmetic (a few FLOPs per element)
is free either way: 32 × 4,096 × 16,384 × ~20 FLOPs ≈ 4e10 FLOPs ≈ 0.04 ms at 990 TFLOPS
fused into the GEMM epilogue: the GEMM writes its output once through the epilogue, which applies bias, GELU and
dropout in registers before the store → the 8.6 GB of the elementwise pass disappears entirely; the GEMM itself
(2 × 32 × 4,096 × 4,096 × 16,384 ≈ 1.8e13 FLOPs ≈ 25 ms at 70% of peak) is unchanged
sanity: unfused, the elementwise chain was 8.6 ms against a 25 ms GEMM (the scenario's 40%/60%); fused into the
epilogue, the block is 25 ms, and the GEMM is all that is left to optimize.
Kernel Fusion has the mechanism; Memory-Bound vs Compute-Bound Kernels and Arithmetic Intensity by Operation are why elementwise ops sit on the bandwidth roof.
Launch overhead, the second saving:
per kernel launch: ~3 to 5 µs of CPU-side work and ~2 to 4 µs of GPU-side scheduling gap between dependent kernels
a decode step for a 7B model at batch 1: ~32 layers × ~12 kernels = ~400 launches → 1.5 to 2 ms of overhead against
~5 ms of actual work; fusing to ~4 kernels per layer or capturing the step in a CUDA graph removes most of it
a training step on a 4 GB activation: the same 400 launches are 2 ms against 500 ms of work; irrelevant
rule: launch overhead matters when kernels are shorter than ~10 µs each; fusion and CUDA graphs are the two fixes,
and they compose
torch.compile and CUDA Graphs covers the graph capture side.
The three fusion shapes:
1. elementwise chains: any sequence of per-element ops (add, mul, activation, cast, dropout) → one kernel that loads
each element once, applies the chain in registers, stores once; torch.compile's Inductor does this automatically
by generating a Triton kernel per chain
2. reductions with their consumers: softmax = max-reduce, subtract-exp, sum-reduce, divide → one kernel per row
holding the row in registers or shared memory (three passes over the row in on-chip memory, one pass over HBM);
layernorm the same (mean, variance, normalize, scale, shift in one kernel); a fused softmax reads the logits once
instead of four times
3. GEMM epilogues: bias, activation, residual add, dtype cast, even a row-wise scale for quantization, applied in the
GEMM's output stage before the store; CUTLASS and cuBLASLt expose this; the savings are the whole elementwise pass
plus the intermediate tensor's memory
plus the big one: attention. FlashAttention fuses QKᵀ, softmax and the PV product so the N × N score matrix never
exists in HBM; that is fusion of a GEMM, a reduction and a GEMM, and it changed the memory cost of attention from
quadratic to linear in sequence length
Triton Programming Model is how most of these are written today; FlashAttention Internals is the attention case; CUTLASS and Tensor Core Kernels is the epilogue mechanism.
What fusion cannot fix:
compute-bound operations: a large GEMM at 70% of peak is limited by tensor-core throughput; fusing anything into it
saves the other thing's traffic, not the GEMM's time; the GEMM's remaining 30% is tiling, pipelining and layout
fusions that serialize: fusing a reduction across a dimension that the unfused version parallelized can cut
occupancy; a fused layernorm over a 16k-wide row with one block per row needs enough rows to fill the GPU
register pressure: a long fused chain holds many live values per thread; past the register budget the compiler spills
to local memory and the fused kernel loses to the unfused one; the fix is splitting the chain or reducing the
per-thread tile
data-dependent control flow and graph breaks: a fusion needs a static chain; a Python-side branch or a host sync
breaks the graph and the fuser starts over at the next segment
cross-kernel reuse the cache already gives: two kernels touching a 20 MB tensor that fits L2 may already hit L2 on
the second pass; fusion's gain is smaller there than the HBM arithmetic suggests
sanity: the test for whether fusion will help is the roofline position of the ops being fused; bandwidth-bound and
launch-bound ops gain, compute-bound ops do not.
Roofline Model is that test; Occupancy and Register Pressure is the spill limit.
The reversal condition: on a system where the elementwise ops are already fused by the compiler and the GEMMs dominate, the next win is not more fusion but precision (FP8 for the GEMMs) and tiling; the candidate who reaches for fusion when the profile shows 90% GEMM time has not read the profile. Nsight Compute's memory chart shows whether the fused kernel actually stopped touching HBM, which is the only proof that the fusion did what you intended.
What interviewers probe next
- "How does torch.compile decide what to fuse?" Inductor groups consecutive pointwise and reduction ops with compatible shapes into one Triton kernel, up to a size limit, and keeps GEMMs and convolutions as library calls with fused epilogues where the backend supports them.
- "Fused softmax on a 128k-wide row?" The row does not fit one block's registers; the kernel uses shared memory or two passes with an online formulation (the same trick FlashAttention uses); still one HBM pass.
- "What is the memory saving beyond bandwidth?" The intermediate tensors do not exist, so peak memory drops; for the MLP example, 4.3 GB per intermediate at batch 32.
- "Can fusion make numerics worse?" It can make them different: a fused kernel may accumulate in a different order or precision; keep fp32 accumulation inside fused reductions and test against the unfused reference.
Common mistakes
- Fusing into a compute-bound GEMM and expecting the GEMM to speed up.
- A fused chain so long it spills registers.
- Counting FLOPs to estimate elementwise cost instead of bytes.
- Forgetting launch overhead as the reason small-batch decode is slow.
Key takeaways
- Elementwise ops cost bytes; N unfused ops move 2N passes of the tensor; fused, 2; a 3× gain for three ops.
- Launch overhead (a few µs per kernel) dominates when kernels are under ~10 µs; fuse and use CUDA graphs.
- Shapes: elementwise chains, reduction-plus-consumers, GEMM epilogues; attention is all three at once.
- Fusion cannot speed up a compute-bound GEMM, and it fails when it spills registers or serializes parallelism.