TL;DR: Measure the shared fraction first, because it decides the size of the prize. An agent request typically carries a system prompt, a tool schema and a conversation history, and if two thousand of a request's twenty-two hundred prompt tokens are shared, prefix caching removes 91 percent of the prefill work. Both engines implement prefix caching. SGLang's RadixAttention organizes cached prefixes in a radix tree so partial and branching prefixes share cleanly, which suits agents where many requests share a long head and diverge at the tail. vLLM's prefix caching is effective and its ecosystem, tooling and model coverage are broader. The practical differentiators are three: how the cache evicts under memory pressure, whether the engine supports your model's exact architecture on day zero, and which tuning model your team can operate, since SGLang publishes numeric targets that make tuning a closed loop while vLLM's tuning is more open-ended. Choose on those, and validate with your own traffic rather than a benchmark's.
How to approach it
Compute what prefix caching is worth on your traffic before comparing engines, since a low sharing rate makes the whole comparison irrelevant. Then the three differentiators. Then say how you would decide empirically, which is a replay of real traffic against both. Close with the factor that overrides all of it, which is model support.
A strong answer
A typical situation: an agent product sends a 2,000-token system prompt and tool schema with every request, followed by a conversation of a few hundred tokens. Traffic is 40 requests per second. The team is choosing an engine and has read that one of them is better for shared prefixes.
What caching is worth, on this traffic:
per request
shared prefix 2,000 tokens (system prompt and tool definitions)
unique tail 200 tokens (the user's turn)
total prompt 2,200 tokens
prefill work without caching, on a 70B-class model
FLOPs = 2 x params x tokens = 2 x 70e9 x 2,200 = 3.08e14
with the shared prefix cached
only the 200 unique tokens are prefilled
FLOPs = 2 x 70e9 x 200 = 2.80e13
saved = 91 percent of prefill work
at 40 requests per second
saved per second = 40 x (3.08e14 - 2.80e13) = 1.12e16 FLOPs/s
at 40 percent of a 9 PFLOPS FP8 part: 3.6e15 FLOPs/s per GPU
so the saving is worth roughly 3.1 GPUs of prefill capacity, continuously
sanity: three GPUs of capacity from a feature both engines have is the reason to measure the
sharing rate rather than to argue about engines, and if the sharing rate were 10
percent instead of 91 the entire comparison would be about something else
SGLang Server Arguments That Matter covers the RadixAttention side. vLLM Server Arguments That Matter covers the equivalent flags and the memory model.
The three differentiators:
| Differentiator | What it means | Which way it points |
|---|---|---|
| Cache structure and eviction | A radix tree shares partial and branching prefixes naturally; eviction policy decides what survives under pressure | SGLang's structure suits deep branching from a common head, which is the agent shape |
| Day-zero model support | New architectures need kernels, and sometimes a new cache manager | Check per model; both projects move fast and neither is universally ahead |
| Tuning model | SGLang publishes numeric targets: 100 to 2,000 queued requests, token usage above 0.9, 5 to 8 GB free after startup | A closed loop with a stopping rule is easier for a small team to operate |
the eviction question, which is the one that bites in production
a cache under memory pressure evicts, and what it evicts decides the hit rate
the shared 2,000-token prefix should never be evicted, because every request needs it
a least-recently-used policy keeps it naturally, since it is touched constantly
the risk is a long tail of per-conversation prefixes crowding it out
what to measure
the prefix cache hit rate, which both engines report
the hit rate specifically on the shared head, if you can instrument it
the correlation between hit rate and the KV pool's utilization: a hit rate that falls as
the pool fills is an eviction problem, and the fix is a larger pool or a lower
max_num_seqs rather than a different engine
sanity: a falling hit rate under load is the failure mode to watch for, and it is an
operational problem in either engine rather than a reason to switch
How to decide empirically:
the experiment, which takes a day
1. capture a representative window of production requests, with their real prompts
2. replay against both engines at the same concurrency sweep, on the same hardware, with
the same model and quantization
3. record per level: TTFT p50 and p99, output tokens per second per user, aggregate
throughput, and the prefix cache hit rate
4. compare curves, not peaks, and identify the operating point where each meets the SLO
5. record every non-default flag on both sides
what invalidates the result
replaying the same request repeatedly, which measures the cache rather than the engine
different quantization on the two sides
different concurrency
sanity: the replay has to preserve the real prefix-sharing structure, because a synthetic
workload with a fixed prompt overstates caching enormously and one with unique
prompts understates it to zero
Serving Benchmarks That Do Not Lie covers the methodology in more depth.
The reversal condition: model support overrides everything above. If only one engine supports the model's attention design and quantization format on day zero, that is the engine, and the comparison is a decision for the next model rather than this one. This happens regularly with new architectures, where a hybrid attention stack needs a cache manager that handles both paged blocks and fixed recurrent state, and the project that implements it first is the only option for weeks. Planning for this means being able to run both, which argues for keeping the deployment's engine choice as a configuration rather than as an architecture commitment.
What interviewers probe next
- "How do you measure the sharing rate?" Hash prefixes of production prompts at a few lengths and compute the fraction of tokens covered by a prefix seen recently. It takes an afternoon and it decides the whole question.
- "What if the sharing rate is low?" Then prefix caching is not the differentiator and the choice is made on model support, throughput and operational fit instead.
- "Does caching change the answer's content?" No; it reuses computed keys and values for identical prefixes, so the result is the same up to the numerical variation any batching change causes.
- "What about multi-turn?" Each turn extends the previous prefix, so the whole conversation history is a cache hit and only the new turn is prefilled, which compounds the saving over a session.
Common mistakes
- Choosing an engine on a feature both have without measuring how much of your traffic uses it.
- Benchmarking with a repeated prompt, which measures the cache and reports an unattainable number.
- Ignoring cache eviction, which is where the hit rate goes under production memory pressure.
- Treating the engine choice as permanent when model support changes it per release.
- Comparing peak throughput rather than the operating point where each engine meets the SLO.
Key takeaways
- Measure the prefix-sharing rate first: 2,000 shared of 2,200 prompt tokens removes 91 percent of prefill work, worth about 3.1 GPUs of capacity at 40 requests per second.
- Both engines cache prefixes; the differentiators are cache structure and eviction, day-zero model support, and the tuning model.
- SGLang publishes numeric tuning targets, which makes tuning a closed loop with a stopping rule.
- Decide by replaying real traffic against both at the same concurrency, model and quantization, and compare curves rather than peaks.
- Model support on day zero overrides the comparison, so keep the engine a configuration choice rather than an architectural commitment.