Why do serving engines capture decode steps into CUDA graphs, and what does capture require of the rest of the code?

At batch 1 a decode step is a few milliseconds of GPU work behind several hundred kernel launches, and the CPU cannot issue them fast enough. The arithmetic showing the GPU starve, what replay changes, and the four constraints capture imposes on memory.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

⚡ Kernels & Compilers🔒 Premium

torch.compile and CUDA Graphstorch.compile captures Python into a graph with Dynamo, fuses it into Triton kernels with Inductor, and can wrap the result in a CUDA graph so a whole forward pass is one launch. CUDA graphs are what make batch-1 decode fast in every serving engine, and graph breaks, recompiles and static-shape rules are what make both bite in production. Interviewers ask when compile helps, when it hurts, and how you would know.

Foundational

⚡ Kernels & Compilers

CUDA Programming ModelCUDA splits a program into a host that allocates, copies and enqueues work, and a device that runs thousands of identical threads organized as a grid of blocks. Getting the split right, and knowing that a launch returns before the kernel runs, decides whether your first live-coding kernel produces a correct number or a silent zero.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputBecause decode reads every weight once per step, its speed is a division: memory bandwidth over bytes per step. That one formula gives single-stream tokens per second for any model on any card, the batch curve that flattens at the ridge point, the effect of quantization, and the point where the KV cache rather than the weights becomes the thing being read. This page derives it, works it for a 70B model on four accelerators, and shows how to read a vendor throughput claim against it.

Foundational

🚀 Inference & Serving

Prefill vs DecodeAn LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackSketch a paged attention kernel. What changes from FlashAttention once the KV cache is not contiguous?Next in this trackcuBLAS already gives you a fast GEMM. What does CUTLASS give you that it does not, and when is that worth it?Next in this trackA take-home gives you a working layernorm kernel at a tenth of memory bandwidth. Make it fast and justify every change.

FEDITOR'S NOTE

Scored on the CPU-versus-GPU time arithmetic that makes launch overhead the bottleneck at small batch, on the static-address and no-sync constraints of capture, and on the shape bucketing that follows from fixed graphs.

DISCUSSION · 0

No comments yet — be the first to share your approach.