SGLang Server Arguments That Matter
SGLang's tuning model is different from vLLM's in one way that matters: it exposes the scheduler's aggressiveness and the static memory fraction as direct knobs, and its own documentation gives target values for the runtime signals those knobs move. That makes tuning it a measurement loop rather than guesswork. Aim for a queue of a hundred to a couple of thousand requests, token usage above 0.9, and five to eight gigabytes of free GPU memory after startup, then adjust the flags that move each one.
TL;DR: Tune against three published targets rather than against intuition. SGLang's documentation says to keep the request queue in the range of 100 to 2,000, to aim for token usage above 0.9, and to leave 5 to 8 GB of free GPU memory after startup. Each target has a flag that moves it.
--mem-fraction-staticsets the fraction of GPU memory given to weights plus the KV pool, and it is raised in increments of 0.01 until memory runs out, then backed off.--schedule-conservativenesscontrols how eagerly the scheduler admits requests, lowered toward 0.3 when the cache sits idle with work waiting and raised toward 1.3 when the pool keeps filling.--chunked-prefill-sizeis reduced to 4096 or 2048 when prefill runs out of memory, and--max-running-requestsis reduced when decode does. RadixAttention prefix caching is on by default and is the reason to pick SGLang for workloads with shared prefixes.
The tuning loop, as the project describes it
signals to watch in the server log, with the project's own targets
queue depth (#queue-req) target 100 to 2,000
persistently 0 means the client is not offering enough work,
not that the server is fast
token usage target above 0.9
low usage with requests waiting means the scheduler is being
too cautious
available_gpu_mem at startup target 5 to 8 GB
more than that is unused capacity; less risks running out
the flags that move each
available_gpu_mem <- --mem-fraction-static, raised 0.01 at a time until it fails
token usage <- --schedule-conservativeness, to about 0.3 to admit more eagerly
pool overflowing <- --schedule-conservativeness, to about 1.3 to admit more cautiously
prefill OOM <- --chunked-prefill-size to 4096 or 2048
decode OOM <- --max-running-requests down
stability over all <- --mem-fraction-static to 0.8 or 0.7, trading concurrency for safety
sanity: every one of these is a closed loop with an observable, which is why SGLang tuning
converges quickly and vLLM tuning often does not, and it is worth saying so in an
interview because it is a real difference between the two
What --mem-fraction-static actually covers
the definition, from the project's documentation
--mem-fraction-static = (model weights + KV cache pool) / GPU memory capacity
so it is not the same thing as vLLM's --gpu-memory-utilization
vLLM's fraction is what the engine claims before subtracting weights
SGLang's fraction includes the weights inside the numerator
worked: a 70B model in FP8 on one 288 GB B300
weights = 70 GB
at --mem-fraction-static 0.85: 288 x 0.85 = 245 GB for weights plus pool
KV pool = 245 - 70 = 175 GB
at the corpus figure of 320 KB per token for a 70B grouped-query model:
175e9 / 327,680 = about 534,000 tokens of cache
at 8,192 tokens per sequence: 65 concurrent sequences at full length
and 288 - 245 = 43 GB left, well above the 5 to 8 GB target, so this is under-allocated
and the fraction should go up
sanity: the 5 to 8 GB target is what turns "is my configuration right" into a measurement,
and 43 GB free means roughly 130,000 tokens of cache being left unused
RadixAttention, and when it is the reason to choose SGLang
where the gain is, as arithmetic
an agent workload: 2,000-token system prompt and tool definitions, then a 200-token turn
without prefix caching, prefill per request = 2,200 tokens
with the shared prefix cached, prefill per request = 200 tokens
prefill work saved = 2,000 / 2,200 = 91%
at a 70B model and 2 x params x tokens FLOPs:
saved per request = 2 x 70e9 x 2,000 = 2.8e14 FLOPs
sanity: the gain scales with the shared fraction, so it is enormous for agents and multi-turn
chat and near zero for a workload of unique long documents, which is exactly the
criterion for choosing this engine
The flags, grouped
| Group | Flag | Note |
|---|---|---|
| Parallelism | --tp-size (--tensor-parallel-size), default 1 | Same divisibility constraint as any engine |
| Parallelism | --moe-dp-size (--moe-data-parallel-size), default 1 | Mixture-of-experts data-parallel width |
| Memory | --mem-fraction-static | Weights plus KV pool as a fraction of GPU memory |
| Memory | --max-total-tokens | Hard bound on the token pool |
| Batching | --chunked-prefill-size, -1 disables | Lower it on prefill out-of-memory |
| Batching | --max-running-requests | Lower it on decode out-of-memory |
| Scheduler | --schedule-conservativeness | 0.3 to admit eagerly, 1.3 to admit cautiously |
| Cache | --radix-eviction-policy, default lru | Prefix cache eviction |
| Cache | --disable-radix-cache | Debugging only; turning this on in production discards the main advantage |
| CUDA graphs | --cuda-graph-max-bs-decode | Raise to 512 or 768 at large tensor parallel degrees, at a memory cost |
What interviewers are listening for
The targets, quoted as targets. An engineer who says "I tune --mem-fraction-static up in
steps of 0.01 until the log shows about 5 to 8 GB of available_gpu_mem" is describing a
procedure with a stopping rule, which is what the question is actually asking for. The second
signal is knowing that SGLang's fraction includes the weights and vLLM's does not, because
carrying a number across from one engine to the other is a common and confusing mistake. The
third is naming the workload shape that makes RadixAttention worth it, since choosing an
engine for a feature the traffic cannot use is the more expensive error.
Key takeaways
- Tune against three published targets: 100 to 2,000 queued requests, token usage above 0.9, and 5 to 8 GB free after startup.
--mem-fraction-staticis (weights plus KV pool) over GPU memory, which is a different definition from vLLM's utilization fraction.--schedule-conservativenesstoward 0.3 admits more eagerly; toward 1.3 admits more cautiously when the pool keeps filling.- Prefill out-of-memory is fixed with
--chunked-prefill-sizeat 4096 or 2048; decode out-of-memory with--max-running-requests. - RadixAttention saves the shared fraction of prefill, which is over 90 percent for an agent with a 2,000-token shared prefix and near zero for unique documents.