TL;DR: Fix four variables and report a curve. The input distribution comes from production logs, meaning prompt and output lengths sampled from real traffic with the real prefix-sharing rate, because a fixed prompt pair measures something no user will experience. The concurrency is a sweep rather than a point, since tokens per second at batch one and at batch 128 are different measurements and both are legitimate. The cache state is stated explicitly, cold or warmed to a realistic hit rate, because replaying one prompt with prefix caching enabled measures the cache. And the metric is named precisely: per-user output rate, aggregate throughput, time to first token at a percentile, or goodput counted only from requests that met their target. The deliverable is aggregate throughput plotted against p99 latency, with the operating point marked where it crosses the service level objective. Add the computed bandwidth bound and the achieved fraction, which turns a number into a diagnosis of where the remaining headroom is.
How to approach it
Start from the decision the benchmark has to support, because that determines which metric is primary. Then fix the four variables and say how. Then the sweep and the curve. Then the bound. Close with everything the report has to carry so a reader six months later can tell whether it still applies.
A strong answer
A typical situation: a team reports that a new model achieves 8,400 tokens per second and recommends it. The number was measured with one repeated prompt, prefix caching on, at whatever concurrency the load generator defaulted to, and it does not predict anything about production.
The four variables:
| Variable | The loose version | The version that supports a decision |
|---|---|---|
| Input distribution | One prompt and output length | Lengths sampled from production logs, with percentiles reported |
| Concurrency | One value | A sweep of at least eight levels through and past the intended operating point |
| Cache state | Whatever happened | Stated: cold, or warmed to the production prefix-sharing rate |
| Metric | "Tokens per second" | Named: per-user output rate, aggregate throughput, p99 TTFT, or goodput |
sampling the input distribution
take a window of production requests: prompt token counts and completion token counts
report p50, p90, p99 for both
replay with those distributions rather than their means, because the tail is what sets p99
latency and a mean-only replay never produces one
why the mean is not enough
a workload with p50 800 and p99 12,000 prompt tokens behaves nothing like one with every
prompt at 1,400, even if the means match
the long prompts occupy prefill capacity and delay decode for everyone sharing the step
sanity: replaying the distribution rather than the mean is the difference between a benchmark
that predicts p99 and one that predicts only p50
The sweep and the curve:
per concurrency level, record
time to first token, p50 and p99
per-user output tokens per second
aggregate output tokens per second
goodput: aggregate throughput counted only from requests meeting the SLO
what the curve looks like
aggregate throughput rises with concurrency, then flattens
p99 latency rises, slowly then sharply
the operating point is where p99 crosses the SLO, and capacity is the request rate there
worked, illustrative
concurrency 8: aggregate 1,900 tok/s, p99 TTFT 340 ms, per-user 238 tok/s
concurrency 32: aggregate 5,600 tok/s, p99 TTFT 720 ms, per-user 175 tok/s
concurrency 64: aggregate 7,800 tok/s, p99 TTFT 1,450 ms, per-user 122 tok/s
concurrency 96: aggregate 8,400 tok/s, p99 TTFT 3,100 ms, per-user 88 tok/s
SLO: p99 TTFT under 1,000 ms
operating point: concurrency 32 to 48, so capacity is about 5,600 to 6,500 tok/s
sanity: the headline 8,400 figure is at a concurrency whose p99 is three times the SLO, so
quoting it describes a configuration that would never be run
Serving Benchmarks That Do Not Lie covers the methodology. Latency Metrics: TTFT, TPOT and Goodput covers the metric definitions.
The bound, which makes the result diagnostic:
compute it alongside
bound = aggregate memory bandwidth / active weight bytes per token
achieved fraction = measured aggregate tokens/s at the operating point / bound
why it belongs in the report
a result at 15 percent of the bound says there is a large amount of headroom and points at
launch gaps, collectives or scattered reads
a result at 65 percent says the deployment is close to physical limits and further tuning
will return little
without it, no reader can tell those two apart
sanity: the bound costs two numbers from the model card and the hardware specification, and
it changes the report from "here is a number" to "here is where the remaining
headroom is"
The report's contents, which is what makes it reusable:
- Engine and version, model and revision, quantization format. All three change the result and all three move.
- Hardware, GPU count and parallel degrees, plus every flag that differs from the default.
- The input distribution with its percentiles, and the prefix-sharing rate.
- The full curve, not only the operating point, so a different SLO can be answered from the same data.
- The bound and the achieved fraction.
- The warmup discarded and the run length, so the percentiles are trustworthy.
The reversal condition: if the decision is between two engines or two configurations rather than about absolute capacity, the absolute numbers matter far less than holding everything else identical. There the discipline is a controlled comparison: same model revision, same quantization, same hardware, same input distribution, same concurrency levels, and the only difference is the thing under test. A comparison that changes two variables answers nothing, and it is the most common way an engine benchmark ends up misleading, because quantization differences in particular are easy to introduce without noticing.
What interviewers probe next
- "How long do you run each level?" Long enough for p99 to stabilize, after discarding a warmup window. Report both durations.
- "What if you cannot get production traffic?" Approximate the distribution from whatever logs exist and say the assumption explicitly, because the alternative is a fixed prompt that measures nothing.
- "Why goodput?" It collapses the curve to one honest number by counting only throughput that met the target, which is what capacity planning needs.
- "What invalidates a comparison?" Any variable changing alongside the one under test, and quantization is the one that slips in most often.
Common mistakes
- Reporting peak aggregate throughput, which occurs at a concurrency whose latency nobody would accept.
- Replaying one prompt with prefix caching on, which measures the cache.
- Using mean prompt lengths rather than the distribution, so p99 is never exercised.
- Omitting the bound, so a reader cannot tell a tuning problem from a physical limit.
- Comparing two engines with different quantization, which measures the format rather than the engine.
Key takeaways
- Fix four variables: input distribution from logs, a concurrency sweep, a stated cache state, and a named metric.
- The deliverable is aggregate throughput against p99 latency with the SLO crossing marked, not a peak.
- In the worked example the 8,400 tokens per second headline sits at a p99 three times the SLO, and real capacity is about 5,600 to 6,500.
- Report the bandwidth bound and the achieved fraction, which distinguishes remaining headroom from a physical limit.
- For a comparison, hold everything identical except the variable under test, and watch for quantization slipping in.