What does torch.compile actually do to your model, and when does it fail to help?

Four stages: a bytecode interpreter captures a graph with guards, an autograd pass splits forward from backward, a compiler emits Triton for everything but the matmuls, and a mode that replays the step as one launch. What each buys, and the two failures that silently give it back.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

⚡ Kernels & Compilers🔒 Premium

torch.compile and CUDA Graphstorch.compile captures Python into a graph with Dynamo, fuses it into Triton kernels with Inductor, and can wrap the result in a CUDA graph so a whole forward pass is one launch. CUDA graphs are what make batch-1 decode fast in every serving engine, and graph breaks, recompiles and static-shape rules are what make both bite in production. Interviewers ask when compile helps, when it hurts, and how you would know.

Foundational

⚡ Kernels & Compilers

Kernel FusionAn elementwise or reduction kernel does a few FLOPs per byte and runs at HBM speed, so a chain of five of them costs five trips through HBM for work that needs one. Fusion collapses the chain into a single kernel that keeps intermediates in registers. It is the first lever for anything memory-bound, and knowing what it cannot fix (weight reads in decode, the GEMMs themselves) is what the interview is really testing.

Advanced

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodePrefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.

Advanced

🚀 Inference & Serving🔒 Premium

Serving Engines: vLLM, SGLang and TensorRT-LLMThree engines serve most open-weight models in production, and they converged on the same mechanisms (paged KV, continuous batching, chunked prefill, prefix caching, speculation, disaggregation) while differing in what they optimize first. vLLM is the default for breadth and hardware coverage, SGLang leads on prefix reuse and structured generation, TensorRT-LLM squeezes the most from NVIDIA hardware at the cost of a compile step. This is a dated page, September 2026; the decision table is what to carry into the room, not the version numbers.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhy do serving engines capture decode steps into CUDA graphs, and what does capture require of the rest of the code?Next in this trackSketch a paged attention kernel. What changes from FlashAttention once the KV cache is not contiguous?Next in this trackcuBLAS already gives you a fast GEMM. What does CUTLASS give you that it does not, and when is that worth it?

FEDITOR'S NOTE

Scored on naming the four stages and what each contributes, on knowing that graph breaks and recompiles are the common failures, and on how to diagnose both rather than guessing.

DISCUSSION · 0

No comments yet — be the first to share your approach.