TL;DR: Single-stream decode reads every weight once per token, so the ceiling is bandwidth ÷ weight bytes: 3.35 TB/s ÷ 141 GB ≈ 24 tokens per second for Llama 3.1 70B in bf16 on one H100's worth of bandwidth. Spreading the weights over 8 cards with tensor parallelism gives up to 8x that in principle, about 190, before communication overhead; fp8 weights double it again.
How to approach it
Ask whether "single user" means batch size one, which it usually does, and which precision the weights are in. Then say the mechanism before the number: at batch one, each generated token requires streaming all the weights from HBM through the chip, and the arithmetic per token (2N FLOPs) is far too small to keep the tensor cores busy, so the step time is the weight read time. Compute bytes per step, divide bandwidth by it, and sanity-check against the compute ceiling to show the bound is memory, not math.
A strong answer
A typical situation: a demo streams noticeably slower than a person reads and the team reaches for a faster card. Single-stream decode is a division of bandwidth by bytes, and the tensor cores do not appear in it at all.
A decode step for one sequence multiplies a single token's vector through every weight matrix. The FLOPs are 2N ≈ 1.4e11, which the H100 could do in 0.14 ms at peak; the bytes are all 141 GB of weights, which the H100 needs 42 ms to read. The step is bandwidth-bound by a factor of 300, so throughput is a bandwidth division.
inputs: weights (bf16) = 70.6e9 × 2 B = 141.2 GB
HBM bandwidth (H100 SXM) = 3.35 TB/s
batch = 1, so one token per step
step time = bytes per step ÷ bandwidth
= 141.2e9 ÷ 3.35e12
= 0.0421 s = 42 ms
tokens per second = 1 ÷ step time = 3.35e12 ÷ 141.2e9 ≈ 23.7 → about 24 tokens/s
compute check: 2N = 1.41e11 FLOPs per token ÷ 989e12 FLOP/s = 0.14 ms,
300x shorter than the 42 ms memory time, so memory is the bound.
sanity: 24 tokens/s is roughly 18 words a second, faster than reading speed, which is
why a single-user 70B on H100s feels responsive despite the small number.
One card cannot hold the 141 GB, so this is a per-card-bandwidth figure. In a tensor-parallel deployment over 8 H100s each card reads its 17.6 GB shard per step, which takes 5.3 ms, and the cards then exchange partial results through all-reduces. Ideal scaling gives 8 × 24 ≈ 190 tokens/s; the all-reduces (two per layer, 160 per step, each a few tens of microseconds at batch one) and kernel launch overhead bring measured single-stream TP8 numbers to well under that in practice, often 80 to 120, which is why "up to 190" is the right phrasing and a measured figure beats the estimate when you have one.
The two levers, both of which change the bytes per step:
| change | bytes per step | tokens/s per card-bandwidth |
|---|---|---|
| bf16 weights | 141 GB | 24 |
| fp8 weights | 71 GB | 47 |
| int4 weights | 35 GB | 95 |
| bf16 on H200 (4.8 TB/s) | 141 GB | 34 |
| bf16 on B200 (8 TB/s) | 141 GB | 57 |
Quantization is a pure bandwidth win at batch one, which is why it is the first thing a latency-focused serving team does. The other lever, speculative decoding, drafts several tokens with a small model and verifies them in one big-model step, so each 42 ms read yields two or three accepted tokens instead of one; it changes tokens per step rather than bytes per step.
The reversal condition is batch. Add 32 concurrent users and the same 141 GB read serves 32 tokens, so the per-card throughput rises toward 32 × 24 ≈ 760 tokens/s while each user still sees about 24. Decode stays memory-bound until the batch reaches the ridge point, around 300 on an H100 in bf16, which is the batch curve question. The Bandwidth-Bound Decode Throughput concept carries the full curve. Bandwidth-Bound Decode Throughput is this same division generalized, and nvidia-smi dmon showing memory near saturation with low SM activity is what it looks like on a live card. Model Memory Footprint gives the bytes in the numerator.
What interviewers probe next
- "Why not use the TFLOPS number?" Because 2N FLOPs per token is 0.14 ms of tensor-core time against 42 ms of HBM time; the tensor cores idle 99.7% of the step at batch one.
- "What does the user experience?" Time per output token of 42 ms, so a 500-token reply streams in 21 seconds on one card's bandwidth, or 4 to 6 seconds on TP8.
- "How does the KV cache change this?" It adds bytes per step: at 8k context the cache is 2.6 GB, 2% of the weights, negligible at batch one; at batch 64 the cache reads exceed the weight reads.
- "What about an 8B model?" 16 GB in bf16 ÷ 3.35 TB/s ≈ 210 tokens/s on one card, the same formula.
Common mistakes
- Dividing peak TFLOPS by FLOPs per token and reporting 7,000 tokens/s for a single user, a number only reachable at large batch.
- Forgetting that a single H100 cannot hold the bf16 model, and reporting 24 tokens/s as a one-card deployment.
- Assuming TP8 gives a clean 8x; communication and launch overheads take a large fraction at batch one.
- Confusing per-user speed with fleet throughput when batch is greater than one.
Key takeaways
- Single-stream decode tokens/s = bandwidth ÷ weight bytes: 3.35 TB/s ÷ 141 GB ≈ 24 for a bf16 70B.
- The step is memory-bound by about 300x at batch one; TFLOPS do not enter.
- Halve the bytes (fp8: 47 tokens/s, int4: 95) or raise the bandwidth (H200: 34, B200: 57) to speed one user up.
- Batch multiplies fleet throughput without changing per-user speed until the ridge, around batch 300.