The engine logs say CUDA graphs are disabled for your deployment. Does it matter?

Graphs remove the per-kernel launch cost, which is a large share of a decode step at low batch and a small one at high batch, so whether it matters is a question about your operating point. The arithmetic, the four reasons an engine turns them off, and the measurement that settles it.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

⚡ Kernels & Compilers🔒 Premium

torch.compile and CUDA Graphstorch.compile captures Python into a graph with Dynamo, fuses it into Triton kernels with Inductor, and can wrap the result in a CUDA graph so a whole forward pass is one launch. CUDA graphs are what make batch-1 decode fast in every serving engine, and graph breaks, recompiles and static-shape rules are what make both bite in production. Interviewers ask when compile helps, when it hurts, and how you would know.

Core

🚀 Inference & ServingSign in

Continuous BatchingContinuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputBecause decode reads every weight once per step, its speed is a division: memory bandwidth over bytes per step. That one formula gives single-stream tokens per second for any model on any card, the batch curve that flattens at the ridge point, the effect of quantization, and the point where the KV cache rather than the weights becomes the thing being read. This page derives it, works it for a 70B model on four accelerators, and shows how to read a vendor throughput claim against it.

Foundational

🧮 Open Weights & Serving Engines

Serving Benchmarks That Do Not LieMost published serving numbers are not comparable to each other and not predictive of production, because they differ in the input distribution, the concurrency, whether the cache was warm, and which of several very different metrics is being reported. A benchmark that supports a decision has to fix all four, report a distribution rather than a mean, and be run against the traffic shape you actually serve. The single most useful discipline is to compute the bandwidth bound first, so you know what fraction of the possible you achieved.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackFifty teams have fine-tuned the same base model. How do you serve all of them?Next in this trackA replica restarts and loads different weights than its siblings. How did that happen?Next in this trackYou must serve a frontier open-weights model on non-NVIDIA accelerators. Plan it.

FEDITOR'S NOTE

Scored on launch overhead being a fraction that shrinks with batch size, on the reasons graphs get disabled, and on measuring the gap rather than assuming it.

DISCUSSION · 0

No comments yet — be the first to share your approach.