mediumNewLambdaRunPodNebius

A startup wants to train a 7B model on 1 trillion tokens in three weeks. What cluster do they rent?

6ND to a compute budget, the fleet equation to a GPU count, the rental rate to a bill: 64 H100s, about 19 days, roughly $75k plus a buffer. The chain, the reasons the answer is 64 and not 59, and the memory check that says the fleet is compute-sized, not memory-sized.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core

🧮 Napkin Math & CapacitySign in

GPU-Hours and Time to TrainThe fleet equation turns a training run's FLOPs into a schedule: time = 6ND divided by (GPUs times peak FLOPS times MFU). Every term is a stated assumption, and the interviewer grades the assumptions rather than the digits: which peak, which MFU, and what happens to the answer when MFU falls from 40% to 30%. This page works three runs end to end (an 8B, a 70B and a 405B), inverts the equation for the GPU count a deadline needs, and shows the sensitivity that separates a considered estimate from a lucky one.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputBecause decode reads every weight once per step, its speed is a division: memory bandwidth over bytes per step. That one formula gives single-stream tokens per second for any model on any card, the batch curve that flattens at the ridge point, the effect of quantization, and the point where the KV cache rather than the weights becomes the thing being read. This page derives it, works it for a 70B model on four accelerators, and shows how to read a vendor throughput claim against it.

Foundational

🧮 Napkin Math & Capacity

KV Cache SizingThe KV cache is the memory that decides how many users a serving replica can hold and how long their context can be. Its size per token comes from four numbers in the model's config file (layers, KV heads, head dimension, bytes per element) and one formula; multiplied by context and concurrency it is the number every capacity plan is built on. This page derives it, works it for four models including an MLA one, and shows the two places candidates get it wrong by a factor of eight.

Foundational

🕸️ Distributed Training

Data Parallelism and DDPData parallelism gives every GPU a full copy of the model, feeds each a different slice of the batch, and averages the gradients with an all-reduce so every replica takes the same optimizer step. It is the first parallelism every training job uses, and the tokens-per-GPU arithmetic behind it decides whether the communication hides behind the backward pass or dominates the step.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackEstimate the activation memory for training a 70B model at an 8k sequence length Next in this trackModel our serving request queue with Little's law. What happens as we approach saturation?Next in this trackRank the H100, MI300X and Trainium2 by cost per token for decode

FEDITOR'S NOTE

The interviewer wants the smallest fleet that meets the deadline with stated MFU, rounded to whole nodes, plus a budget with a buffer and the memory check. A candidate who quotes GPUs without a node count or a calendar has not planned a rental.

DISCUSSION · 0

No comments yet — be the first to share your approach.