mediumNewFireworksTogether AI

How much does moving from bf16 to fp8 save in serving cost?

fp8 halves the bytes per decode step and doubles the compute peak, but the all-reduces and launch overhead do not shrink, so the measured gain is 1.5x at a fixed batch and about 2x once the freed memory raises the batch. The chain, with the step-time table.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputBecause decode reads every weight once per step, its speed is a division: memory bandwidth over bytes per step. That one formula gives single-stream tokens per second for any model on any card, the batch curve that flattens at the ridge point, the effect of quantization, and the point where the KV cache rather than the weights becomes the thing being read. This page derives it, works it for a 70B model on four accelerators, and shows how to read a vendor throughput claim against it.

Foundational

🧮 Open Weights & Serving Engines

Weight Formats: FP8 Blocks, MXFP4 and AWQOpen-weights models now ship pre-quantized, and the format is part of the release rather than something you choose afterwards. Block-scaled FP8 gives one byte per parameter with a scale per tile. MXFP4 gives about 0.53 bytes by pairing four-bit values with a shared exponent every 32 elements. Integer schemes like AWQ reach similar sizes with a different error profile. What decides a deployment is not which is most accurate in the abstract but which one the model was released and evaluated in, and which one your engine and hardware can execute natively.

Core

🧮 Napkin Math & CapacitySign in

Arithmetic Intensity by OperationThe roofline says a kernel's ceiling is set by its FLOPs per byte against the hardware's ridge point. This page does the FLOPs-per-byte arithmetic for the operations an LLM actually runs (decode at several batch sizes, prefill, the attention score matmul with and without FlashAttention, LayerNorm, an embedding lookup) so the reader can place any of them on the roofline from first principles and say which lever moves it. The numbers explain why a serving fleet's GPUs report 30% utilization while fully loaded.

Advanced

🚀 Inference & Serving🔒 Premium

Quantization for InferenceQuantization stores weights, and sometimes activations and the KV cache, in fewer bits, which cuts the bytes a decode step has to stream and the memory a model occupies. Weight-only int4 (GPTQ, AWQ) is a capacity and single-stream latency play; fp8 for weights and activations (W8A8) doubles tensor-core throughput and helps prefill and large batch; fp8 KV cache doubles context per GPU. Each has an accuracy cost you measure rather than assume, and knowing which one to reach for from the bottleneck is the interview question.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow long does an all-reduce of a 70B model's gradients take on eight GPUs?Next in this trackHow many tokens per second can one B200 decode for a 70B model?Next in this trackWhat storage throughput does a 16k-GPU training cluster need?

FEDITOR'S NOTE

The interviewer wants the two mechanisms named (bytes per step, compute peak) and then the honest measured multiplier, with the overhead term that keeps it below 2x. The candidate who says 'fp8 halves the cost' without the overhead has not run it.

DISCUSSION · 0

No comments yet — be the first to share your approach.