What is MFU, how do you compute it from a running job, and what counts as a good number?

TL;DR: MFU is the FLOPs the model mathematically needs, 6 × parameters × tokens, divided by what the GPUs could have done in the same wall-clock time, GPUs × peak × seconds. On H100s a well-tuned large dense run lands at 35 to 45%; Llama 3 405B reported 380 to 430 TFLOPS per GPU, which is 38 to 43% of the 989 TFLOPS dense bf16 peak. The rest goes to communication that is not overlapped, pipeline bubbles, memory-bound kernels, and recompute, in roughly that order.

How to approach it

Define it in one sentence, then immediately write the formula with units. Say what goes in the numerator (model FLOPs, 6ND, never the FLOPs the hardware executed) and what goes in the denominator (dense peak, never the sparsity-doubled marketing number). Compute it for a concrete step. Then give the band, cite the Llama 3 anchor, and list what eats the remainder, each with the metric that would reveal it. Close with why MFU and not "GPU utilization" from nvidia-smi.

A strong answer

A typical situation: a run of a 70B model reports 6.2 seconds per step on 512 H100s with a global batch of 4 million tokens, and the team wants to know whether that is good. The question has a numeric answer.

inputs:  N = 70.6e9 parameters
         tokens per step T = 4.0e6
         step time = 6.2 s
         GPUs g = 512, dense bf16 peak = 989 TFLOPS = 989e12 FLOP/s

model FLOPs per step = 6 × N × T
                     = 6 × 70.6e9 × 4.0e6 = 1.69e18 FLOPs

available FLOPs      = g × peak × step time
                     = 512 × 989e12 × 6.2 = 3.14e18 FLOPs

MFU = 1.69e18 ÷ 3.14e18 = 0.54 → 54%

sanity: 54% on H100s for a 70B is above the band a frontier run reports, so either the
        step time is optimistic (measured without the dataloader and the optimizer) or the
        batch size is being counted after padding. Check both before celebrating.

The 6 in the numerator is 2 FLOPs per parameter per token for the forward pass (one multiply, one add per weight) and 4 for the backward pass (gradient with respect to the input and gradient with respect to the weight). It counts only the matrix multiplies of the parameters; attention's score computation, which scales with sequence length squared, is left out of the standard definition and adds a few percent at 8k context and a lot at 128k. MFU and HFU covers the difference between this model-FLOPs number and hardware FLOPs, which includes recompute.

The denominator uses the dense peak. NVIDIA's headline H100 figure is 1,979 TFLOPS for bf16 with 2:1 structured sparsity, which no training GEMM reaches. Dividing by it halves every MFU number and makes an ordinary run look like a disaster.

What is a good number depends on the generation. On A100s, well-tuned large dense runs reported 50 to 60% (PaLM on TPU v4 reported 46%; Megatron-LM on A100s reported over 50% for the largest models). On H100s the band moved down to 35 to 45% because peak compute grew about 3× while HBM bandwidth grew 1.6× and NVLink 1.5×, so more kernels are memory-bound and more communication is exposed relative to compute. Meta's Llama 3 report gives 380 to 430 TFLOPS per GPU at 8k and 16k context for the 405B on 16,384 H100s:

MFU = 400 TFLOPS ÷ 989 TFLOPS ≈ 0.40

That is the anchor to carry. A 70B dense run at 40 to 45% is well tuned; 30% means something is wrong that can be found; 20% means several things are wrong.

Where the other 60% goes, with the signal for each:

• Communication that is not hidden behind compute. TP all-reduces that overlap poorly, the last DDP bucket, FSDP all-gathers that arrive late. Visible in a profiler trace as NCCL kernels with no overlapping compute, and in the gap between HFU and MFU being small while step time is long.
• Pipeline bubbles. With p stages and m micro-batches the bubble is (p−1)/m of compute time; PP16 with 64 micro-batches idles about 19% of every step. Visible as per-stage idle time in the trace.
• Memory-bound kernels. Layernorm, softmax, residual adds, the optimizer step, and attention at long context all run far below peak because they stream bytes. DCGM_FI_PROF_PIPE_TENSOR_ACTIVE well below DCGM_FI_PROF_SM_ACTIVE is the shape.
• Recompute. Activation checkpointing re-runs forward, which raises hardware FLOPs by about a third without adding model FLOPs. This is the item that HFU counts and MFU does not.
• Stragglers and jitter. The collective waits for the slowest rank; a 2% slower GPU costs the whole job 2%.
• Everything outside the step. Dataloader stalls, checkpoint writes, evaluation. These lower the MFU averaged over a day even when the per-step number is fine.

The decision this number drives: below about 35% on a dense H100 run, stop scaling out and profile, because adding GPUs to a run that is exposing communication makes the fraction worse. The condition that reverses the target is architecture. A mixture-of-experts run with all-to-all traffic and a long-context run with quadratic attention have lower achievable MFU by construction, and the comparison has to be against a peer run, not against the dense band.

The reversal condition: an MoE, where 6ND uses active parameters and the same wall-clock step produces a much lower MFU that is not a defect. Training FLOPs: 6ND states which N belongs in the numerator, and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE is the field to cross-check the step-time estimate against.

What interviewers probe next

• "Why not just read GPU utilization from nvidia-smi?" That column reports the fraction of time any kernel was resident, so a memory-bound kernel at 5% of peak shows 100%; MFU measures useful work against the ceiling.
• "Should MFU include the attention FLOPs?" State the convention. Most reports use 6ND; at 128k context the attention term is comparable to the linear term for a 70B, and a report that includes it will look better than one that does not.
• "MoE MFU: active or total parameters?" Active, because only active parameters do multiplies; a DeepSeek-V3-style model at 37B active on 671B total reports MFU against 37B.
• "What would raise this run from 40 to 45%?" Larger micro-batch to amortize communication, fused kernels for the memory-bound layers, and fewer pipeline stages with more micro-batches; each is worth a percent or two and there is no single 5% item left in a tuned run.

Common mistakes

• Dividing by the sparse peak of 1,979 TFLOPS and reporting 20% for a healthy run.
• Reporting HFU (which counts recompute) as MFU, which inflates the number by up to a third.
• Measuring the step time in a warm loop with no dataloader, no logging and no checkpointing, and then being surprised when the daily average is 30% lower.
• Using the tokens per step before dropping padding, which inflates the numerator.

Key takeaways

• MFU = 6 × N × tokens ÷ (GPUs × dense peak × seconds); every term has a unit, and the peak is the dense one.
• H100 frontier band: 35 to 45%. Llama 3 405B at 400 TFLOPS per GPU is 40%.
• The missing 60% is exposed communication, bubbles, memory-bound kernels, recompute and stragglers, each with a metric that shows it.
• Below 35% on a dense run, profile before adding GPUs.