GPUs and time-to-train calculator

Q: What MFU should I assume?

Frontier dense runs on H100 fleets report 35 to 45%; well-tuned Blackwell and TPU runs land in a similar band. Anything above 55% needs an explanation. State the number you used, then show how the answer moves if it is 10 points lower.

Q: Why does the memory floor use 16 bytes per parameter?

Mixed-precision Adam keeps bf16 weights (2 bytes), bf16 gradients (2), fp32 master weights (4), and two fp32 optimizer moments (8). Activations are extra and depend on batch, sequence length and checkpointing, so 16 bytes per parameter is the floor, not the total.

Estimate training FLOPs with 6ND, wall-clock time on a fleet at a stated MFU, the GPU count for a deadline, the fleet cost, and the minimum GPUs needed to hold mixed-precision Adam state. Every assumption is visible.

Run

ParametersBTraining tokensT

Model FLOPs utilization: 40%40%Frontier dense runs report 35 to 45% on H100 fleets. Say which end you assumed.

Fleet

Accelerator

GPUsDeadlinedays$ per GPU-hourUSD

Training compute

6300000000.00PFLOPs

6 × 70B × 15T

Wall clock on 4,096 GPUs

45.0 days

3.9M tokens/s sustained

GPUs for a 30-day deadline

6,144

at 40% MFU on H100 80GB SXM

Fleet cost

$11.06M

4.42M GPU-hours × $2.5/h

Does the static state fit?

Mixed-precision Adam holds 16 bytes per parameter before a single activation: bf16 weights and gradients, fp32 master weights and two fp32 moments. For 70B that is 1.1 TB, which fully sharded (ZeRO-3 / FSDP) needs at least 18 × H100 80GB SXM at 80% usable memory. Activations, the allocator and communication buffers come on top, so treat this as the floor the interviewer expects you to state, not the fleet you would run.

How this was computed

FLOPs: 6 × 70e9 × 15e12 = 6300000000.00P FLOPs
Effective rate: 4,096 × 989 TFLOPS × 40% = 1620.38P FLOP/s
Time: FLOPs ÷ effective rate = 45.0 days
GPUs for deadline: FLOPs ÷ (989e12 × 40% × 30 d) = 6,144
Static memory: 70e9 × 16 B = 1.1 TB

What the interviewer is actually asking

"How many H100s would you need to train a 70B model on 15 trillion tokens in a month?" is asked at almost every lab and cloud. The number matters less than the shape: 6ND, a peak you name, an MFU you defend, and the memory floor from optimizer state that decides the minimum fleet before speed enters into it.

Questions people ask

Why 6 FLOPs per parameter per token?▲

Two for the forward pass (a multiply and an add per weight), and four for the backward pass, which computes gradients with respect to both activations and weights. Attention over the sequence adds a term that grows with context length, but for typical pre-training contexts it is a small correction to 6ND.

What MFU should I assume?▼

Why does the memory floor use 16 bytes per parameter?▼