Why is fp8 training hard, and how did DeepSeek-V3 make it work?

Three mantissa bits and a range of 448 make fp8 unusable with one scale per tensor. The arithmetic of what an outlier channel destroys, the 128-element block scaling and fp32 promotion that the DeepSeek-V3 report used, the ops that stayed in bf16, and what the 2× peak bought in practice.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core

🧩 GPU & Accelerator ArchitectureSign in

Numerics: FP32, BF16, FP8 and FP4Every number format is a trade between range (exponent bits), precision (mantissa bits) and throughput (fewer bits, more values per cycle through the tensor cores). bf16 won training because it keeps fp32's range; fp8 splits into E4M3 for precision and E5M2 for range and needs scaling factors; fp4 needs block scaling and careful outlier handling. Knowing which format goes where, and why accumulation stays fp32, is what the numerics question is really asking.

Advanced

🕸️ Distributed Training🔒 Premium

Expert Parallelism for MoEA mixture-of-experts layer runs only a few of its experts per token, so the experts can be spread across GPUs and each token shipped to the ranks that hold its chosen experts. That shipping is an all-to-all in each direction, twice per layer per pass, and its cost plus the load imbalance between experts is what expert parallelism is really about.

Foundational

🧮 Open Weights & Serving Engines

Open-Weights Models of 2026The open-weights frontier moved from dense models of tens of billions of parameters to sparse mixtures of experts measured in trillions, and the serving problem changed with it. As of September 2026 the releases an infrastructure engineer is asked about are Z.ai's GLM-5.3 at 753B, Moonshot's Kimi K3 at 2.8T, and DeepSeek's V4 family. What matters for deployment is not the headline count but three other numbers: active parameters per token, the attention design, and the format the weights actually shipped in.

Advanced

⚡ Kernels & Compilers🔒 Premium

CUTLASS and Tensor Core KernelsCUTLASS is NVIDIA's template library for building GEMM-shaped kernels that run tensor cores at near cuBLAS speed while letting you change the data types, the tile shapes and the epilogue. Its hierarchy (device, kernel, collective mainloop, tile, instruction) is the vocabulary of every tensor-core discussion, and knowing when it beats calling cuBLAS or writing Triton is the judgment question kernel interviews end on.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign a checkpoint format for thousands of GPUs: no gather on write, resumable at a different world size, no stall.Next in this trackYour 4,096-GPU run loses about 2% of every day to restarts. Fix it, and tell me where the floor is.Next in this trackPlan a 10-trillion-token pre-training run end to end: compute, fleet, layout, data, checkpoints and a schedule with a failure budget.

FEDITOR'S NOTE

Scored on deriving why per-tensor scaling fails from the format's range, on knowing the two mechanisms of the recipe (block scales and periodic fp32 promotion), and on listing what stayed in higher precision and why.

DISCUSSION · 0

No comments yet — be the first to share your approach.