In an int8 or fp8 GEMM, what exactly gets scaled, where does the dequantization happen, and what breaks if you get it wrong?

The matmul runs on tiny formats and the accumulator does not, so every low-precision GEMM is really a scaled integer or float product with a correction applied on the way out. Where the scales come from, the three granularities and what each costs, and why one of them needs a periodic promotion into fp32.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core

🧩 GPU & Accelerator ArchitectureSign in

Numerics: FP32, BF16, FP8 and FP4Every number format is a trade between range (exponent bits), precision (mantissa bits) and throughput (fewer bits, more values per cycle through the tensor cores). bf16 won training because it keeps fp32's range; fp8 splits into E4M3 for precision and E5M2 for range and needs scaling factors; fp4 needs block scaling and careful outlier handling. Knowing which format goes where, and why accumulation stays fp32, is what the numerics question is really asking.

Foundational

🧮 Open Weights & Serving Engines

Weight Formats: FP8 Blocks, MXFP4 and AWQOpen-weights models now ship pre-quantized, and the format is part of the release rather than something you choose afterwards. Block-scaled FP8 gives one byte per parameter with a scale per tile. MXFP4 gives about 0.53 bytes by pairing four-bit values with a shared exponent every 32 elements. Integer schemes like AWQ reach similar sizes with a different error profile. What decides a deployment is not which is most accurate in the abstract but which one the model was released and evaluated in, and which one your engine and hardware can execute natively.

Advanced

🚀 Inference & Serving🔒 Premium

Quantization for InferenceQuantization stores weights, and sometimes activations and the KV cache, in fewer bits, which cuts the bytes a decode step has to stream and the memory a model occupies. Weight-only int4 (GPTQ, AWQ) is a capacity and single-stream latency play; fp8 for weights and activations (W8A8) doubles tensor-core throughput and helps prefill and large batch; fp8 KV cache doubles context per GPU. Each has an accuracy cost you measure rather than assume, and knowing which one to reach for from the bottleneck is the interview question.

Foundational

🧮 Napkin Math & Capacity

Training FLOPs: 6NDThe compute needed to train a language model is six floating-point operations per parameter per token: two for the forward pass and four for the backward. Multiply by the parameter count and the token count and you have the whole run's compute, which is the number every fleet-sizing, time-to-train and cost question starts from. This page derives the 6, states the attention correction and when it matters, and shows where the 2N of inference comes from, so the reader can rebuild the formula rather than recall it.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackA mixture-of-experts layer is 256 small matmuls. Why is a loop over cuBLAS calls slow, and what does a grouped GEMM do instead?Next in this trackSelect the top k logits from a 128,000-token vocabulary on the GPU. What shape does the kernel take and why not just sort?Next in this trackYou added three local variables to a working kernel and it got 30 percent slower. Explain what happened and how you would confirm it.

FEDITOR'S NOTE

Scored on knowing dequantization belongs in the epilogue, on the granularity trade-off with the outlier argument, and on the accumulator precision issue that forces periodic promotion in fp8 training.

DISCUSSION · 0

No comments yet — be the first to share your approach.