Core

GPU-Hours and Time to Train

The fleet equation turns a training run's FLOPs into a schedule: time = 6ND divided by (GPUs times peak FLOPS times MFU). Every term is a stated assumption, and the interviewer grades the assumptions rather than the digits: which peak, which MFU, and what happens to the answer when MFU falls from 40% to 30%. This page works three runs end to end (an 8B, a 70B and a 405B), inverts the equation for the GPU count a deadline needs, and shows the sensitivity that separates a considered estimate from a lucky one.

a free account unlocks the core curriculum tier · no card

RELATED CONCEPTS

Training FLOPs: 6ND MFU and HFU Model Memory Footprint Cost per Million Tokens TCO: Buy vs Rent Elastic and Fault-Tolerant Training

PRACTICE THIS IN REAL QUESTIONS

Napkin Math, Cost & CapacityHow many H100s do you need to train a 70B model on 15 trillion tokens in 30 days?→Napkin Math, Cost & CapacityHow long does that 70B run take on 16,384 H100s at 40% MFU?→Napkin Math, Cost & CapacityEstimate how many H100-hours it took to train Llama 3.1 405B, then check it against the paper→Napkin Math, Cost & CapacityA startup wants to train a 7B model on 1 trillion tokens in three weeks. What cluster do they rent?→Napkin Math, Cost & CapacityWhat does one training token cost?→Napkin Math, Cost & CapacitySize an inference fleet for a 70B model serving 1,000 concurrent users→