Set max-model-len and max-num-seqs for a chat product from first principles.

TL;DR: Derive both from the product, not the model. For --max-model-len, take the context length distribution from production logs and set it at a high percentile plus a margin, because the flag determines the worst-case cache a single sequence can reserve and the model's maximum is usually far above what anything sends. For --max-num-seqs, compute two ceilings and take the lower. The memory ceiling is the KV pool divided by the product of max-model-len and KV bytes per token. The latency ceiling comes from the SLO: every sequence in a batch waits for the whole batch's step, so per-user output rate falls roughly as concurrency rises, and there is a batch size beyond which the time per output token misses its target. Most deployments are limited by the second and configured as if they were limited by the first. Verify by comparing the engine's reported KV pool in tokens against your arithmetic, then run a concurrency sweep and set the flag at the SLO crossing rather than at the memory limit.

How to approach it

Get the context distribution from logs before touching anything, since it sets the first flag. Compute the memory ceiling for the second. Then compute the latency ceiling from the SLO, which requires a sweep rather than arithmetic. Take the lower. Verify against the engine's own reporting. Close with what to re-check when traffic changes.

A strong answer

A typical situation: a chat product is deployed with both flags at their defaults. The model supports 1,048,576 tokens of context, the product's p99 request uses 6,200, and the engine has reserved cache for a context nothing will ever send while running out of room for concurrency.

The first flag, from the traffic:

--max-model-len, from production logs
  measure prompt + expected output tokens per request, and take percentiles
  example distribution for a chat product:
    p50   1,400 tokens
    p90   3,800
    p99   6,200
    p999  9,500
  set max-model-len at p999 plus a margin: 12,288
  requests above it are rejected with a clear error rather than silently truncated, so the
    percentile chosen is a product decision as well as a capacity one

what the model's maximum would have cost
  reserved worst case per sequence at 1,048,576 tokens against 12,288 is 85 times more
sanity: the flag bounds the worst case per sequence, so setting it from the model rather than
        from the traffic is the single most expensive default in a serving configuration

The second flag, memory ceiling:

KV pool per GPU
  usable = GPU memory x --gpu-memory-utilization
  pool   = usable - weights per GPU - activation and context headroom

  worked, a 70B model in FP8 at TP=4 on 80 GB parts, utilization 0.92:
    usable  = 80 x 0.92 = 73.6 GB
    weights = 70 / 4    = 17.5 GB
    headroom, activations and context: 8 GB
    pool per GPU = 48.1 GB, so 192.4 GB across the four

memory ceiling on concurrency
  KV per token at the corpus figure for a 70B grouped-query model: 320 KB
  tokens of cache = 192.4e9 / 327,680 = 587,000
  sequences at max-model-len 12,288 = 587,000 / 12,288 = 47.8, so 47
sanity: the memory ceiling here is 47, which is far below the default max_num_seqs, and a
        deployment left at the default will simply never reach it because the pool runs out
        first and the scheduler queues

The second flag, latency ceiling:

why concurrency costs latency
  a decode step produces one token per sequence in the batch
  the step time grows with batch size, slowly at first and then faster as compute binds
  per-user time per output token = step time, so it rises with concurrency

the sweep that finds it
  run concurrency 1, 2, 4, 8, 16, 32, 48 and record
    step time, hence per-user tokens/s
    aggregate tokens/s
    p99 time to first token
  find the largest concurrency where per-user time per output token still meets the SLO

illustrative
  SLO: 40 ms per output token, so 25 tokens/s per user
  measured per-user rate: 62 at concurrency 8, 41 at 24, 26 at 40, 19 at 48
  latency ceiling = 40 concurrent sequences
sanity: the two ceilings here are 47 from memory and 40 from latency, so the flag is set to
        40 and the memory has headroom, which is the common case and the opposite of how the
        flag is usually chosen

vLLM Server Arguments That Matter covers the flags and the memory model. Latency Metrics: TTFT, TPOT and Goodput covers the target the second ceiling is measured against.

Verification:

compare arithmetic against the engine
  the engine logs its KV cache size in blocks and in tokens at startup
  your prediction: pool bytes / KV bytes per token
  agreement within a few percent means the model, dtype and attention design were read right
  a large discrepancy means one of those is wrong, and it is worth resolving before tuning
    anything else

then in production
  the fraction of steps at the concurrency cap, which says whether the flag binds
  the KV pool utilization, which says whether memory binds
  p99 time per output token against the SLO
sanity: if the pool utilization is low and the concurrency cap is hit constantly, the flag is
        set below what the memory allows and there is throughput being left unclaimed

The reversal condition: for a batch or offline workload with no latency target, the latency ceiling disappears and --max-num-seqs should be pushed to the memory ceiling and beyond, since aggregate throughput is the only objective and per-user rate is irrelevant. There --max-model-len can also be set to whatever the longest document requires without concern for reserving cache, because there is no concurrent interactive traffic competing for it. The configuration for an offline summarization pipeline and for an interactive chat product on the same model are in every respect different, and running one configuration for both means one of the two workloads is served badly.

What interviewers probe next

• "What happens to requests above --max-model-len?" They are rejected. Choosing the percentile is therefore a product decision about how many users see an error.
• "Why not just raise the utilization fraction?" It gives the profiling run and the CUDA context less headroom, and past a point it causes startup failures rather than more cache.
• "How often do you re-check?" Whenever the traffic shape changes. A product that starts producing longer answers changes both ceilings without any configuration change.
• "What if the two ceilings are far apart?" If memory is the lower one, the deployment needs more GPUs or a smaller context; if latency is lower, there is spare memory that could support a longer context or a bigger cache for prefix reuse.

Common mistakes

• Setting --max-model-len from the model's maximum rather than the product's distribution.
• Setting --max-num-seqs from memory alone and missing the SLO by a wide margin at peak.
• Never running a concurrency sweep, so the latency ceiling is unknown.
• Not comparing the engine's reported cache size against the arithmetic, which is the check that the model was understood.
• Using one configuration for both interactive and offline workloads on the same model.

Key takeaways

• Set --max-model-len from the production context distribution at a high percentile plus margin; the model's maximum can be 85 times larger.
• --max-num-seqs has two ceilings: memory, from the KV pool divided by max-model-len times KV bytes per token, and latency, from the SLO.
• In the worked example the ceilings are 47 from memory and 40 from latency, so the latency one binds, which is the common case.
• Find the latency ceiling with a concurrency sweep, not with arithmetic, because step time against batch size is measured.
• Verify by comparing the engine's reported KV pool in tokens against your prediction; a large gap means a config field was misread.