AI Infra Interviews logo
← Napkin Math, Cost & Capacity / 12
mediumNewMeta

Estimate how many H100-hours it took to train Llama 3.1 405B, then check it against the paper

6ND on 15.6 trillion tokens gives 3.8e25 FLOPs; converting to GPU-hours needs an MFU, and Meta's published 30.8 million hours lets you solve for it. The chain, the check, and what the implied MFU tells you.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

6ND on 15.6 trillion tokens gives 3.8e25 FLOPs; converting to GPU-hours needs an MFU, and Meta's published 30.8 million hours lets you solve for it. The chain, the check, and what the implied MFU tells you.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core
🧮 Napkin Math & CapacitySign in
GPU-Hours and Time to TrainThe fleet equation turns a training run's FLOPs into a schedule: time = 6ND divided by (GPUs times peak FLOPS times MFU). Every term is a stated assumption, and the interviewer grades the assumptions rather than the digits: which peak, which MFU, and what happens to the answer when MFU falls from 40% to 30%. This page works three runs end to end (an 8B, a 70B and a 405B), inverts the equation for the GPU count a deadline needs, and shows the sensitivity that separates a considered estimate from a lucky one.
Core
🕸️ Distributed TrainingSign in
MFU and HFUModel FLOPs utilization is the fraction of a GPU's peak that goes into the model's own forward and backward math, computed from 6ND and the step time; hardware FLOPs utilization also counts recomputation. Production LLM training lands at 35 to 45% MFU, and knowing where the other 55% goes is the job.
Foundational
🧮 Napkin Math & Capacity
Training FLOPs: 6NDThe compute needed to train a language model is six floating-point operations per parameter per token: two for the forward pass and four for the backward. Multiply by the parameter count and the token count and you have the whole run's compute, which is the number every fleet-sizing, time-to-train and cost question starts from. This page derives the 6, states the attention correction and when it matters, and shows where the 2N of inference comes from, so the reader can rebuild the formula rather than recall it.
Core
🚀 Inference & ServingSign in
Attention Variants: MHA, GQA, MQA and MLAThe KV cache scales with the number of key-value heads, and the four attention variants differ exactly there: multi-head keeps one KV head per query head, multi-query keeps one for all, grouped-query shares one across a group, and multi-head latent attention caches a compressed latent instead of keys and values at all. For Llama 3.1 70B that is the difference between 2.6 MB and 320 KB per token; for DeepSeek-V3 it is about 70 KB. The variant a model was trained with is a serving decision made before the first GPU was bought.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackSize an inference fleet for a 70B model serving 1,000 concurrent usersNext in this trackA MoE model and a dense model reach the same quality. How do their GPU counts compare for serving and for training?Next in this trackEstimate how long it takes to write a checkpoint for a 405B training run
FEDITOR'S NOTE

This is a calibration question. The interviewer knows the published number and wants to see the candidate derive a figure within 20% of it and then explain the gap with MFU, not hand-wave it.

DISCUSSION · 0

No comments yet — be the first to share your approach.