vLLM crashes with out of memory during startup on a model that should fit. Debug it.

TL;DR: Separate the two cases first. Out of memory during startup means the weights plus the reserved KV pool plus the profiling run exceeded what the fraction allowed, and the engine reports its own accounting in the log, so read that before changing anything. Out of memory during serving usually means something the profiling run did not exercise, such as an unusually long prompt or a multimodal input. For startup there are four levers with clear arithmetic. Lower --max-model-len, which reduces the worst-case cache per sequence and is almost always the largest and cheapest win. Lower --max-num-seqs, which caps concurrency. Lower --gpu-memory-utilization, which is counterintuitive because it gives vLLM less rather than more but avoids the case where the profiling run itself has no headroom. Or set --kv-cache-dtype fp8, halving cache bytes per token. Before touching any of them, do the arithmetic: weights over the parallel degree plus a realistic cache plus about 15 percent, against the memory per GPU, tells you whether the configuration was ever possible.

How to approach it

Read the engine's memory report from the log first, since it states what it thinks the weights, the profile and the cache each cost. Then do the arithmetic independently and compare, because a mismatch identifies which assumption is wrong. Then apply the levers in order of cost. Close with the runtime case, which is a different failure with different causes.

A strong answer

A typical situation: a 70B model in FP8 is launched on 4 GPUs of 80 GB, which is 320 GB against 70 GB of weights, and it fails during startup with an allocation error. The instinct is to raise --gpu-memory-utilization, which makes it worse.

What the memory is actually spent on:

per GPU, at tensor parallel size N
  model weights           total weight bytes / N
  activation workspace    depends on max_num_batched_tokens and the model's width
  CUDA context and
    library workspace     a fixed cost, commonly a few hundred megabytes to a couple of GB
  the profiling run       vLLM runs a forward pass at the largest configured shape to measure
                          peak activation memory, and that pass must fit
  KV cache pool           whatever is left after all of the above, within the utilization
                          fraction

the accounting
  usable = GPU memory x --gpu-memory-utilization
  KV pool = usable - weights/N - activation peak - context
  and if that comes out negative or tiny, startup fails or the engine reports a cache too
    small to serve anything
sanity: the profiling run is the part people do not know about, and it is why a model whose
        weights obviously fit can still fail at startup: the profile runs at
        max_num_batched_tokens and that peak has to fit alongside the weights

vLLM Server Arguments That Matter covers the flags. Reading config.json to Size a Model You Have Never Run covers the arithmetic to compare against.

The independent arithmetic, done before changing flags:

the example: 70B in FP8, TP=4, 80 GB parts, max_model_len left at the model's maximum
  weights per GPU        70 GB / 4 = 17.5 GB
  usable at 0.92         80 x 0.92 = 73.6 GB
  remaining              73.6 - 17.5 = 56.1 GB for activations, context and cache

  now the cache requirement the configuration implies
    max_model_len at the model's maximum, say 131,072
    max_num_seqs default, say 256
    worst case reserved = 256 x 131,072 tokens of cache
    at the corpus 320 KB per token for a 70B grouped-query model:
      256 x 131,072 x 327,680 B = 11.0 TB
  against 4 x 56.1 = 224 GB available
sanity: the configuration asked for 11 TB of cache on a system with 224 GB, so it was never
        going to start, and no flag that adjusts the utilization fraction fixes an
        arithmetic gap of 49 times

The four levers, in order of cost:

applying them to the example
  max_model_len 131,072 -> 8,192:  reserved cache falls by 16x, to 688 GB
  max_num_seqs 256 -> 64:          falls by another 4x, to 172 GB
  now 172 GB against 224 GB available, which starts
  and with --kv-cache-dtype fp8: 86 GB, comfortable
sanity: two flags that reflect what the product actually needs took the requirement from 11 TB
        to 172 GB, which is why the first question is what context and concurrency the
        product requires rather than what the model supports

The runtime case, which is a different failure:

out of memory during serving, after a successful start
  the profiling run measured peak activation memory at the configured shapes
  something exceeded it
    a prompt longer than max_model_len should have allowed, if the limit is not enforced
      upstream
    a multimodal input whose encoder allocates outside the profiled path
    a fragmentation problem after long uptime
    another process on the same GPU

  what to check
    the request that triggered it, from the engine log
    nvidia-smi during steady state, to see whether another process is resident
    whether the failure is reproducible with that exact request
sanity: a startup failure is an arithmetic problem and a runtime failure is a specific
        request, so the log line that names the request is worth more than any amount of
        flag tuning

The reversal condition: if the arithmetic says the configuration fits comfortably and it still fails at startup, the problem is not the configuration. Candidates are another process holding memory on the same GPU, a previous run that did not release memory, an incorrect tensor parallel degree so that each rank loads more than its share, or a quantization config the engine did not apply so that FP8 weights are being loaded as bf16 at twice the size. That last one is worth checking early because it is common and it doubles the weight footprint silently, and nvidia-smi during load plus the engine's reported weight size against your prediction identifies it in one step.

What interviewers probe next

• "Why does raising the utilization fraction sometimes make it worse?" Because the profiling run and the CUDA context need headroom outside the fraction, so a very high value leaves nothing for them.
• "How do you know the reported weight size is right?" Compare the engine's reported figure against total parameters times bytes per parameter from quantization_config. A 2x discrepancy means the quantization was not applied.
• "What is a sensible --max-model-len?" The product's real maximum plus a margin, not the model's. It is the single largest lever in this whole exercise.
• "Would you use swap?" vLLM can offload, and it trades a large latency penalty for capacity. It is a last resort rather than a sizing strategy.

Common mistakes

• Raising --gpu-memory-utilization in response to a startup failure, which removes the headroom the profiling run needs.
• Leaving --max-model-len at the model's maximum, which reserves cache for a context nothing will use.
• Changing flags before doing the arithmetic, so it is unknown whether any configuration fits.
• Not noticing that a FP8 model is being loaded as bf16, which doubles the weights silently.
• Treating a runtime failure as a sizing problem when the log names a specific request.

Key takeaways

• Startup failure is an arithmetic problem; runtime failure is a specific request, and the log distinguishes them.
• The profiling run must fit alongside the weights, which is why a very high utilization fraction can cause the failure.
• Do the independent arithmetic: 256 sequences at 131,072 tokens on a 70B grouped-query model asks for 11 TB of cache.
• Four levers in cost order: --max-model-len, --max-num-seqs, --kv-cache-dtype fp8, then --gpu-memory-utilization.
• If the arithmetic says it fits and it still fails, check for another process, the wrong parallel degree, or a quantization config that was not applied.