Explain mixed-precision training. Why does fp16 need loss scaling and bf16 not, why keep fp32 master weights, and what changes with fp8?

fp16 loses gradients below 6e-8 and overflows above 65,504; bf16 has fp32's range and three digits of precision. The arithmetic that shows why an update of 3e-5 vanishes into a bf16 weight, where each of the 16 bytes per parameter comes from, and what fp8 block scaling adds on top.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core

🧩 GPU & Accelerator ArchitectureSign in

Numerics: FP32, BF16, FP8 and FP4Every number format is a trade between range (exponent bits), precision (mantissa bits) and throughput (fewer bits, more values per cycle through the tensor cores). bf16 won training because it keeps fp32's range; fp8 splits into E4M3 for precision and E5M2 for range and needs scaling factors; fp4 needs block scaling and careful outlier handling. Knowing which format goes where, and why accumulation stays fp32, is what the numerics question is really asking.

Foundational

🧮 Open Weights & Serving Engines

Weight Formats: FP8 Blocks, MXFP4 and AWQOpen-weights models now ship pre-quantized, and the format is part of the release rather than something you choose afterwards. Block-scaled FP8 gives one byte per parameter with a scale per tile. MXFP4 gives about 0.53 bytes by pairing four-bit values with a shared exponent every 32 elements. Integer schemes like AWQ reach similar sizes with a different error profile. What decides a deployment is not which is most accurate in the abstract but which one the model was released and evaluated in, and which one your engine and hardware can execute natively.

Advanced

🚀 Inference & Serving🔒 Premium

Quantization for InferenceQuantization stores weights, and sometimes activations and the KV cache, in fewer bits, which cuts the bytes a decode step has to stream and the memory a model occupies. Weight-only int4 (GPTQ, AWQ) is a capacity and single-stream latency play; fp8 for weights and activations (W8A8) doubles tensor-core throughput and helps prefill and large batch; fp8 KV cache doubles context per GPU. Each has an accuracy cost you measure rather than assume, and knowing which one to reach for from the bottleneck is the interview question.

Foundational

🕸️ Distributed Training

Data Parallelism and DDPData parallelism gives every GPU a full copy of the model, feeds each a different slice of the batch, and averages the gradients with an all-reduce so every replica takes the same optimizer step. It is the first parallelism every training job uses, and the tokens-per-GPU arithmetic behind it decides whether the communication hides behind the backward pass or dominates the step.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYour training run is at 60% of the step time you projected. How do you find out whether it is compute, memory, network or I/O?Next in this trackTraining loss went flat at step 40k after descending normally. Walk me through how you debug it.Next in this trackAt 16,000 GPUs something fails every few hours. How do you choose the checkpoint interval, and what does the write have to look like?

FEDITOR'S NOTE

Scored on knowing the exponent and mantissa widths and deriving the consequences, on explaining master weights from the rounding of a small update, and on placing fp8 as a third tier with its own scaling problem.

DISCUSSION · 0

No comments yet — be the first to share your approach.