mediumNewNVIDIACoreWeave

How many tokens per second can one B200 decode for a 70B model?

8 TB/s over the bytes per step: 57 tokens per second single-stream in bf16, 113 in fp8, 227 in fp4, and the batch curve on a single card that now holds the whole model. Where the curve bends and what caps it.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputBecause decode reads every weight once per step, its speed is a division: memory bandwidth over bytes per step. That one formula gives single-stream tokens per second for any model on any card, the batch curve that flattens at the ridge point, the effect of quantization, and the point where the KV cache rather than the weights becomes the thing being read. This page derives it, works it for a 70B model on four accelerators, and shows how to read a vendor throughput claim against it.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

GPU Generations: A100 to BlackwellFour NVIDIA generations are in fleets at once, and interviewers ask what each one changed, not what it is called. A100 to H100 added fp8 and tripled compute; H200 kept the die and grew memory; B200 doubled everything and added fp4; B300 stacked more HBM and cut fp64. This page carries the dense numbers for each, what they did to training and serving, and the marketing traps (sparse peaks, 192 versus 180 GB, die counting) that trip candidates. Dated September 2026.

Foundational

🖧 Hardware & Cluster Build-Out

Accelerator Selection: H100 to B300 and RTX PRO 6000Three published numbers decide which accelerator suits a workload, and they are independent: memory capacity gates what fits, memory bandwidth gates decode speed, and tensor FLOPS gate prefill and training. As of September 2026 the parts NVIDIA sells for datacenters span 80 GB to 288 GB and 1.6 TB/s to 8 TB/s, and the gap between the compute number and the bandwidth number has widened every generation, which is why a part that looks four times faster on a slide is often twice as fast on a decode workload.

Core

🧮 Napkin Math & CapacitySign in

Arithmetic Intensity by OperationThe roofline says a kernel's ceiling is set by its FLOPs per byte against the hardware's ridge point. This page does the FLOPs-per-byte arithmetic for the operations an LLM actually runs (decode at several batch sizes, prefill, the attention score matmul with and without FlashAttention, LayerNorm, an embedding lookup) so the reader can place any of them on the roofline from first principles and say which lever moves it. The numbers explain why a serving fleet's GPUs report 30% utilization while fully loaded.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhat storage throughput does a 16k-GPU training cluster need?Next in this trackTraffic peaks at three times the daily average. Capacity-plan the serving fleet.Next in this trackWhat does one training token cost?

FEDITOR'S NOTE

The interviewer wants the candidate to build the batch curve from bytes per step, not quote a single number, and to notice that a B200 holds a 70B in fp8 on one card, which changes the deployment shape.

DISCUSSION · 0

No comments yet — be the first to share your approach.