Training loss went flat at step 40k after descending normally. Walk me through how you debug it.

A loss that stops moving has five different causes and one of them is not a training problem at all. The checks in order: is the step counter moving, what are the learning rate and the grad norm doing, is the data repeating, has the loss scaler collapsed, and is one rank sending zeros.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Stragglers and HangsSynchronous training runs at the speed of its slowest rank, so one GPU that is 30% slow makes a thousand GPUs 30% slow, and one rank that never arrives at a collective makes the other 1,023 wait in silence until a watchdog fires ten minutes later. Finding the slow rank and the stuck rank is the most common on-call task on a training fleet, and the tooling for it (per-rank timing, the NCCL flight recorder, stack dumps across ranks) is specific and learnable. This page derives the straggler tax from first principles, lists the causes in the order they actually occur, and gives the procedure for a hang.

Foundational

🧮 Open Weights & Serving Engines

Weight Formats: FP8 Blocks, MXFP4 and AWQOpen-weights models now ship pre-quantized, and the format is part of the release rather than something you choose afterwards. Block-scaled FP8 gives one byte per parameter with a scale per tile. MXFP4 gives about 0.53 bytes by pairing four-bit values with a shared exponent every 32 elements. Integer schemes like AWQ reach similar sizes with a different error profile. What decides a deployment is not which is most accurate in the abstract but which one the model was released and evaluated in, and which one your engine and hardware can execute natively.

Advanced

📐 AI Systems Design🔒 Premium

Evaluation and Data Pipeline InfrastructureBehind every model release is a pipeline that turns raw text into training shards and a harness that runs thousands of evaluation prompts against every checkpoint, and both are infrastructure problems with GPU-sized budgets. The data side is a batch system: dedup, filter, tokenize and shard petabytes with lineage. The eval side is a serving system in disguise: run a benchmark suite against a checkpoint in minutes, on shared GPUs, reproducibly, with results a researcher can trust. This page designs both, derives the compute and storage they need, and gives the reproducibility rules that separate a real harness from a script.

Core

🧩 GPU & Accelerator ArchitectureSign in

Numerics: FP32, BF16, FP8 and FP4Every number format is a trade between range (exponent bits), precision (mantissa bits) and throughput (fewer bits, more values per cycle through the tensor cores). bf16 won training because it keeps fp32's range; fp8 splits into E4M3 for precision and E5M2 for range and needs scaling factors; fp4 needs block scaling and careful outlier handling. Knowing which format goes where, and why accumulation stays fp32, is what the numerics question is really asking.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackAt 16,000 GPUs something fails every few hours. How do you choose the checkpoint interval, and what does the write have to look like?Next in this trackDesign a training system that survives losing a node without a human in the loop. What does elasticity cost you?Next in this trackOne GPU out of 16,000 is 15% slow and the whole run is 15% slow. How do you find it, and what is usually wrong with it?

FEDITOR'S NOTE

Scored on checking that the run is alive before touching the model, on reading the learning rate and grad norm logs before forming a theory, and on knowing that a resumed dataloader can silently replay data.

DISCUSSION · 0

No comments yet — be the first to share your approach.