hardNewDeepSeekMistralGoogle

A MoE model and a dense model reach the same quality. How do their GPU counts compare for serving and for training?

Memory follows total parameters and compute follows active ones, so the two models pull the GPU count in opposite directions. The chain for a 235B-total, 22B-active MoE against a dense 70B, for serving and then for training.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🕸️ Distributed Training🔒 Premium

Expert Parallelism for MoEA mixture-of-experts layer runs only a few of its experts per token, so the experts can be spread across GPUs and each token shipped to the ranks that hold its chosen experts. That shipping is an all-to-all in each direction, twice per layer per pass, and its cost plus the load imbalance between experts is what expert parallelism is really about.

Core

🧮 Napkin Math & CapacitySign in

GPU-Hours and Time to TrainThe fleet equation turns a training run's FLOPs into a schedule: time = 6ND divided by (GPUs times peak FLOPS times MFU). Every term is a stated assumption, and the interviewer grades the assumptions rather than the digits: which peak, which MFU, and what happens to the answer when MFU falls from 40% to 30%. This page works three runs end to end (an 8B, a 70B and a 405B), inverts the equation for the GPU count a deadline needs, and shows the sensitivity that separates a considered estimate from a lucky one.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputBecause decode reads every weight once per step, its speed is a division: memory bandwidth over bytes per step. That one formula gives single-stream tokens per second for any model on any card, the batch curve that flattens at the ridge point, the effect of quantization, and the point where the KV cache rather than the weights becomes the thing being read. This page derives it, works it for a 70B model on four accelerators, and shows how to read a vendor throughput claim against it.

Foundational

🧮 Napkin Math & Capacity

Model Memory FootprintThe first calculation in almost every AI infra loop: how many bytes does this model occupy, for inference and for training, and does it fit on the card in front of you? Inference is parameters times bytes per parameter (2 in bf16), plus a KV cache that grows with users. Training is 16 bytes per parameter before activations. A 70B model is 141 GB to serve and 1.13 TB to train, and a reader who can produce those two numbers from the parameter count, with the reasoning, has passed the first five minutes of the estimation round.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackEstimate how long it takes to write a checkpoint for a 405B training run Next in this trackWhat is the power budget for a cluster of 1,024 H100s?Next in this trackShould we buy or rent 1,000 H100s for the next two years?

FEDITOR'S NOTE

The interviewer wants the candidate to separate the two parameter counts and to show that the answer flips between serving (memory-bound, MoE needs more cards to hold) and per-token cost (compute and bandwidth per token, MoE wins).

DISCUSSION · 0

No comments yet — be the first to share your approach.