TL;DR: The bandwidth bound says tokens per second is aggregate bandwidth divided by active weight bytes, and it assumes the GPU spends the whole step streaming weights at peak rate. At batch one it does not. Four terms account for the gap. Kernel launch and scheduling overhead, since a step is hundreds of small kernels and at batch one each does very little work, so launch latency is a large fraction. Collective latency, since tensor-parallel all-reduces and expert all-to-alls have a fixed cost per call that does not shrink with batch size. Non-contiguous reads, because a mixture-of-experts model reads a scattered subset of experts rather than one contiguous block, and achieved bandwidth on scattered reads is well below peak. And host-side work, meaning sampling, detokenization and the Python scheduler between steps. Together these routinely leave batch-one decode at 10 to 20 percent of the bound on large sparse models. The two recoveries are batching, which amortizes every fixed cost across more tokens, and speculative decoding, which amortizes them across more tokens without needing more concurrent users.
How to approach it
State the bound and its two assumptions, then break each. Name the four terms specifically rather than calling them overhead. Explain why sparse models are worst affected, since that is the interesting part. Then the two recoveries with the arithmetic for each. Close with what to measure to attribute the gap on a specific deployment.
A strong answer
A typical situation: a team computes that a model should reach about 1,000 tokens per second at batch one on their hardware, measures 111, and concludes the deployment is misconfigured. It is not. The bound was never attainable at batch one, and the useful question is which of four terms to attack.
The bound and its assumptions:
the bound
tokens/s <= aggregate memory bandwidth / active weight bytes per token
for a 104B-active model with expert weights in a four-bit format on 8 parts at 8 TB/s:
active bytes ≈ 63 GB
bound = 64e12 / 63e9 = about 1,020 tokens/s
what it assumes
1. the GPU is reading weights at peak bandwidth for the entire step
2. nothing else in the step costs time
sanity: both assumptions are approximately true at large batch and badly false at batch one,
which is why the bound is a ceiling rather than a prediction
The four terms:
| Term | Why it does not shrink with batch size | Rough scale at batch one |
|---|---|---|
| Kernel launch and scheduling | A step is hundreds of kernel launches; each has a fixed cost regardless of the work inside | Microseconds each, hundreds per step, so a meaningful fraction of a millisecond-scale step |
| Collective latency | An all-reduce or all-to-all has a fixed setup and synchronization cost per call | Two per layer for tensor parallelism, two per mixture-of-experts layer for expert parallelism |
| Non-contiguous weight reads | Selected experts are scattered across memory, so achieved bandwidth is below peak | Achieved bandwidth on scattered reads can be a large fraction below the streaming figure |
| Host-side work | Sampling, detokenization and the scheduler run between steps on the CPU | Sub-millisecond per step, but the step itself is only a few milliseconds |
why sparse models suffer most
a dense model reads its weights as long contiguous runs, so the achieved bandwidth is close
to peak
a mixture-of-experts model at batch one reads 8 of 256 experts, chosen per token, from
wherever they sit in memory
the read is many medium-sized pieces rather than one stream, and the memory system delivers
less than peak on that pattern
meanwhile the total bytes are small, so the fixed costs are a larger share
the compounding
fewer bytes per step -> the step is shorter
shorter step -> fixed costs are a larger fraction
scattered reads -> the bytes that are read move slower than peak
so the two effects multiply rather than add
sanity: this is why a published batch-one figure of about 11 percent of the bound is normal
for a large sparse model and would be alarming for a dense one
Bandwidth-Bound Decode Throughput covers the bound itself. Serving Benchmarks That Do Not Lie covers reporting the achieved fraction alongside any number.
The two recoveries:
recovery 1: batching
every fixed cost is per step, not per token
at batch B, the same weight read serves B tokens
tokens/s ≈ B x (1 / step time), and step time grows only slowly with B until compute or
memory binds
worked, using illustrative round numbers:
batch 1: step 9 ms, 111 tokens/s
batch 32: step 11 ms, 2,909 tokens/s
batch 128: step 18 ms, 7,111 tokens/s
the per-user rate falls as B grows, because each user waits for the whole batch's step
sanity: batching is the primary recovery and it trades aggregate throughput against per-user
latency, which is exactly the curve a concurrency sweep produces
recovery 2: speculative decoding
a draft proposes k tokens, the target verifies them in one forward pass
the fixed costs are paid once for up to k+1 accepted tokens
so it raises per-user throughput at batch one, which batching cannot do
the published example for a large sparse model reports 111 tokens/s rising to 331 with a
speculative configuration, described by the project as 3.14 times
the gain depends on the acceptance rate: at acceptance a and draft length k, the expected
tokens per target step is roughly (1 - a^(k+1)) / (1 - a)
at a = 0.8 and k = 7: (1 - 0.8^8) / 0.2 = (1 - 0.168) / 0.2 = 4.16 tokens per step
sanity: speculation is the recovery that helps a single user, and batching is the one that
helps aggregate throughput, so a product with low concurrency and a latency target
needs the first and a high-volume batch product needs the second
Speculative Decoding covers the acceptance mechanics.
What to measure to attribute the gap:
a step-level profile on one rank
Nsight Systems for a few steps, then attribute:
time in matmul and attention kernels the useful work
time in collectives all-reduce and all-to-all
gaps between kernels launch and scheduling overhead
host time between steps sampling, detokenization, the scheduler
the diagnostic ratios
useful kernel time / step time how close to the bound you could get
gap time / step time what CUDA graphs would recover
collective time / step time what a better all-to-all backend or a
smaller parallel degree would recover
sanity: a step that is 40 percent gaps is a CUDA graph problem, and one that is 40 percent
collectives is a topology or backend problem, and those have completely different
fixes, which is why the profile comes before any tuning
The reversal condition: if the deployment serves many concurrent users, the batch-one number is irrelevant and optimizing for it is the wrong work. At batch 128 the fixed costs are spread over 128 tokens per step and the achieved fraction of the bound rises substantially, so the deployment may already be near its ceiling while its batch-one figure looks poor. Report the achieved fraction at the operating concurrency, which is where the throughput-latency curve crosses the SLO. Quoting a batch-one figure for a high-concurrency product describes a configuration nobody runs.
What interviewers probe next
- "What would CUDA graphs recover?" The launch and scheduling gaps, by replaying a captured sequence of kernels rather than launching each. It is the standard fix for the first term.
- "Why does the acceptance rate matter so much?" Because a rejected draft token wastes the verification work, so the expected tokens per step falls quickly as acceptance drops.
- "Would a smaller tensor-parallel degree help?" Sometimes, since fewer ranks means fewer collectives per layer. It trades against memory and against per-GPU bandwidth.
- "How do you report this honestly?" Give the bound, the measured number, and the achieved fraction at a stated concurrency, so the reader can tell a configuration problem from a physical limit.
Common mistakes
- Treating the bandwidth bound as a target rather than a ceiling.
- Calling the gap "overhead" without naming which of the four terms dominates.
- Optimizing batch-one throughput for a product that runs at high concurrency.
- Assuming a sparse model achieves peak bandwidth, when scattered expert reads do not.
- Enabling speculative decoding without measuring the acceptance rate, which determines whether it helps at all.
Key takeaways
- The bound assumes contiguous streaming at peak and no other cost, and both fail at batch one.
- Four terms: kernel launch gaps, fixed collective latency, non-contiguous expert reads, and host-side sampling and scheduling.
- Sparse models suffer most because fewer bytes per step make fixed costs a larger share while scattered reads lower achieved bandwidth.
- Batching raises aggregate throughput and lowers per-user rate; speculative decoding raises per-user rate at batch one, from a published 111 tokens/s to 331 on one large sparse model.
- Expected tokens per target step under speculation is about (1 - a^(k+1)) / (1 - a), which at acceptance 0.8 and draft length 7 is 4.16.
- Attribute with a step profile: mostly gaps means CUDA graphs, mostly collectives means topology or backend.