TL;DR: Pick by bottleneck. Weight-only int4 (AWQ, GPTQ) cuts a 70B model from 141 GB to about 35 GB and speeds up low-batch decode, so it is for capacity and small fleets; it dequantizes to bf16 for the matmul, so it gives nothing at high batch. fp8 W8A8 halves weights and runs the GEMMs on fp8 tensor cores at 1,979 TFLOPS dense on H100, so it is for throughput at batch. fp8 KV cache halves cache per token and is for context and concurrency. Ship only after task evals on your own traffic, not perplexity alone.
How to approach it
Ask what is short: GPU memory, tokens per second at batch, or concurrent context. Ask the hardware, because fp8 needs Hopper or newer and int4 kernels vary by card. Then map the three methods to the three bottlenecks with the byte and FLOP arithmetic, name the accuracy risks specific to each, and finish with the measurement plan you would run before turning it on for customers.
A strong answer
A typical situation: a team quantizes for throughput, gets capacity instead, and cannot explain why the tokens per second barely moved. They changed the storage precision and not the one the matmul runs in.
Quantization for Inference is three separate decisions that happen to share a word: what precision the weights are stored in, what precision the matmul runs in, and what precision the KV cache is stored in.
inputs: Llama 3.1 70B, H100 SXM (3.35 TB/s, 989 bf16 / 1,979 fp8 TFLOPS dense)
weights
bf16: 70.6e9 × 2 B = 141.2 GB (needs 2 cards minimum, 8 for a KV budget)
fp8: 70.6e9 × 1 B = 70.6 GB (1 card is still tight; 2 cards comfortable)
int4: 70.6e9 × 0.5 B ≈ 35.3 GB (1 card with 40+ GB of KV budget left)
batch-1 decode step ≈ weight bytes ÷ bandwidth (one card, ignoring KV)
bf16: 141.2 ÷ 3.35 ≈ 42 ms (does not fit; shown for scale)
fp8: 70.6 ÷ 3.35 ≈ 21 ms → 47 tokens/s
int4: 35.3 ÷ 3.35 ≈ 10.5 ms → 95 tokens/s
sanity: at batch 1 the step is a weight read, so halving bytes halves the step; int4 wins here
compute ceiling at high batch (tokens/s = peak × MFU ÷ (2 × params)), one card, MFU 0.5
bf16 matmul: 989e12 × 0.5 ÷ 1.41e11 ≈ 3,500 tokens/s
fp8 matmul: 1,979e12 × 0.5 ÷ 1.41e11 ≈ 7,000 tokens/s
int4 weight-only: dequantized to bf16 before the GEMM → the bf16 ceiling, 3,500 tokens/s
sanity: past the ridge, only the matmul precision matters, and weight-only int4 runs bf16 math
That last line is the one candidates miss. Weight-only int4 shrinks bytes and speeds the memory-bound regime; once the batch is large enough to be compute-bound, its dequantization is pure overhead and its throughput ceiling is the bf16 one. fp8 W8A8 (weights and activations both in fp8, scaled per tensor or per block) is the method that raises the compute ceiling, because Hopper's tensor cores run fp8 at twice the bf16 rate.
KV cache quantization is independent of both. Storing K and V in fp8 (with per-head or per-block scales) halves 328 KB per token to 164 KB, which doubles the sequences that fit and shrinks the per-step KV read. It touches the attention kernel, not the linear layers, so it composes with either weight scheme.
| Short on | Method | What it changes | Cost |
|---|---|---|---|
| Memory (fit on fewer cards) | weight-only int4 (AWQ, GPTQ) | 4x smaller weights, faster low-batch decode | dequant overhead at batch; outlier sensitivity; no gain past the ridge |
| Throughput at batch | fp8 W8A8 | 2x weights, 2x matmul peak on Hopper and later | activation outliers need per-block scaling; not on A100 |
| Context and concurrency | fp8 KV cache | 2x cache per token | small attention-score error; needs kernel support |
The accuracy risks differ by method, and perplexity hides most of them. Weight-only int4 concentrates its error in layers with outlier channels; AWQ's activation-aware scaling exists because those channels matter more than their magnitude suggests. fp8 activations fail on the same outliers unless scaling is fine-grained (per 128-element block is the DeepSeek-V3 recipe). KV fp8 degrades long-context retrieval before it degrades short answers, so a needle-in-a-haystack style test at your max context is the one to run.
The measurement plan before shipping:
- Task evals on a sample of production prompts with a judge or exact-match metric, compared to the bf16 baseline, gated at a delta you set (many teams use 1% on their primary eval).
- Long-context and structured-output checks (JSON validity rate, tool-call argument accuracy) where quantization error shows first.
- A throughput and latency run at the production batch on the production kernel, because the kernel, not the format, decides the speedup; an int4 checkpoint on a card without a fused dequant-GEMM can be slower than bf16.
- Watch for the tell in the logs: repeated tokens or degenerate loops at low temperature are how a bad quantization shows up in production before the eval catches it.
Decision: int4 for fitting, fp8 W8A8 for throughput, fp8 KV for context, and fp8 everywhere as the default on H100 and later when the evals pass. Numerics: FP32, BF16, FP8 and FP4 covers what each format can represent, and --kv-cache-dtype fp8 is the one of the three decisions that costs nothing to try. The reversal condition: any measured regression on the customer's own eval set, or a card without fp8 tensor cores (A100), where int8 W8A8 via SmoothQuant is the throughput path instead.
What interviewers probe next
- "Why does int4 need calibration data but fp8 often does not?" fp8 has enough dynamic range that a per-tensor or per-block scale from the weights alone works; int4 has 16 levels, so the rounding decision must be informed by which channels the activations actually stress.
- "Can you serve an int4 model at batch 128?" Yes, but it runs at the bf16 compute ceiling plus dequant cost, so it is slower than fp8 W8A8 on the same card; use it there only if memory forces it.
- "What about fp4 on Blackwell?" B200 lists 9,000 dense fp4 TFLOPS against 4,500 fp8; with block scaling (NVFP4) it is the next throughput step, and the eval gate is the same.
Common mistakes
- Picking int4 for a throughput problem.
- Quoting perplexity as the acceptance test.
- Comparing formats without the kernel in hand.
- Forgetting that the KV cache is a separate quantization decision with its own failure mode at long context.
Key takeaways
- 70B weights: 141 GB bf16, 70.6 GB fp8, 35 GB int4; batch-1 step time scales with those bytes.
- Weight-only int4 dequantizes to bf16 for the matmul, so its high-batch ceiling is the bf16 ceiling.
- fp8 W8A8 doubles the matmul peak on Hopper; fp8 KV halves cache per token independently.
- Ship on task evals, long-context checks and a real throughput run on the production kernel.