Three engines, one hardware roofline, and the difference between them is which part of the roofline each one reaches first on your workload. The decision is a table, and the table has reversal conditions.
more free answers with an account · no card
← LLM Inference & Serving / 15
hardNewBasetenTogether AINVIDIA
vLLM, SGLang or TensorRT-LLM: which engine do you pick for a new deployment, and what would change your mind?
Three engines, one hardware roofline, and the difference between them is which part of the roofline each one reaches first on your workload. The decision is a table, and the table has reversal conditions.
Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.
The concepts behind this question
Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.
Advanced
Serving Engines: vLLM, SGLang and TensorRT-LLMThree engines serve most open-weight models in production, and they converged on the same mechanisms (paged KV, continuous batching, chunked prefill, prefix caching, speculation, disaggregation) while differing in what they optimize first. vLLM is the default for breadth and hardware coverage, SGLang leads on prefix reuse and structured generation, TensorRT-LLM squeezes the most from NVIDIA hardware at the cost of a compile step. This is a dated page, September 2026; the decision table is what to carry into the room, not the version numbers.🚀 Inference & Serving🔒 Premium
Advanced
Chunked PrefillA long prompt's prefill can occupy a GPU for hundreds of milliseconds, and every sequence mid-decode on that GPU waits for it. Chunked prefill splits the prompt into fixed token budgets and interleaves each chunk with a decode step, so decode latency stays flat at the cost of a slower first token for the long prompt. The chunk budget is a knob between TTFT and TPOT, and the interview question is how you would set it.🚀 Inference & Serving🔒 Premium
Advanced
Prefix Caching and KV ReuseMost requests to a production LLM share a prefix: the same system prompt, the same few-shot examples, the same conversation up to the latest turn. Prefix caching keeps the KV blocks for those tokens resident and skips their prefill, so a 4,000-token system prompt costs compute once instead of once per request. Radix trees make the lookup cheap, block-aligned hashing makes it safe, and the hit rate is what decides whether it is a 2x or a 10x win. The interview question is how you would route to make it hit.🚀 Inference & Serving🔒 Premium
Foundational
SGLang Server Arguments That MatterSGLang's tuning model is different from vLLM's in one way that matters: it exposes the scheduler's aggressiveness and the static memory fraction as direct knobs, and its own documentation gives target values for the runtime signals those knobs move. That makes tuning it a measurement loop rather than guesswork. Aim for a queue of a hundred to a couple of thousand requests, token usage above 0.9, and five to eight gigabytes of free GPU memory after startup, then adjust the flags that move each one.🧮 Open Weights & Serving Engines
UP NEXT ON YOUR JOURNEY
Next in this trackHow would you serve hundreds of LoRA adapters on one base model, and what does it cost in throughput?Next in this trackDesign an autoscaler for GPU inference replicas that reacts to load without thrashing.Next in this trackA new replica has to load a 70B model and serve traffic in under a minute. Where do the seconds go, and how do you get there?
FEDITOR'S NOTE
Scored on knowing what each engine optimizes rather than reciting features, on refusing to pick without a workload, and on naming the benchmark you would run before committing.
DISCUSSION · 0
No comments yet — be the first to share your approach.