hardNewCoreWeaveTogether AIBaseten

Compare A100, H100 and B200 for a 70B serving fleet. Which gives the most tokens per dollar, and where do fp8 and fp4 change the ranking?

Decode throughput is bytes per step over bandwidth, price is dollars per hour, so tokens per dollar for a 70B falls out of a spec sheet in ten lines. In bf16 the three generations are within 10% per dollar; fp8 pulls the H100 ahead, and fp4 on the B200 doubles it again. The chain, the table and the caveats.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

GPU Generations: A100 to BlackwellFour NVIDIA generations are in fleets at once, and interviewers ask what each one changed, not what it is called. A100 to H100 added fp8 and tripled compute; H200 kept the die and grew memory; B200 doubled everything and added fp4; B300 stacked more HBM and cut fp64. This page carries the dense numbers for each, what they did to training and serving, and the marketing traps (sparse peaks, 192 versus 180 GB, die counting) that trip candidates. Dated September 2026.

Core

🧩 GPU & Accelerator ArchitectureSign in

Numerics: FP32, BF16, FP8 and FP4Every number format is a trade between range (exponent bits), precision (mantissa bits) and throughput (fewer bits, more values per cycle through the tensor cores). bf16 won training because it keeps fp32's range; fp8 splits into E4M3 for precision and E5M2 for range and needs scaling factors; fp4 needs block scaling and careful outlier handling. Knowing which format goes where, and why accumulation stays fp32, is what the numerics question is really asking.

Foundational

🖧 Hardware & Cluster Build-Out

Accelerator Selection: H100 to B300 and RTX PRO 6000Three published numbers decide which accelerator suits a workload, and they are independent: memory capacity gates what fits, memory bandwidth gates decode speed, and tensor FLOPS gate prefill and training. As of September 2026 the parts NVIDIA sells for datacenters span 80 GB to 288 GB and 1.6 TB/s to 8 TB/s, and the gap between the compute number and the bandwidth number has widened every generation, which is why a part that looks four times faster on a slide is often twice as fast on a decode workload.

Foundational

🧮 Open Weights & Serving Engines

Weight Formats: FP8 Blocks, MXFP4 and AWQOpen-weights models now ship pre-quantized, and the format is part of the release rather than something you choose afterwards. Block-scaled FP8 gives one byte per parameter with a scale per tile. MXFP4 gives about 0.53 bytes by pairing four-bit values with a shared exponent every 32 elements. Integer schemes like AWQ reach similar sizes with a different error profile. What decides a deployment is not which is most accurate in the abstract but which one the model was released and evaluated in, and which one your engine and hardware can execute natively.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackIf you could change one thing about GPU architecture for LLMs, what would it be, and what argues against it?Next in this trackOne node in your cluster shows half the expected NVLink bandwidth in nccl-tests. Walk me through isolating it.Closest to this topicWhat actually changes with Blackwell and the NVL72 rack, and what does it do to how you would serve a large MoE model?

FEDITOR'S NOTE

Scored on running the bandwidth-bound decode chain for each part at a stated batch and context, converting to dollars per million tokens, and correctly identifying that precision, not the generation, is what moves the ranking.

DISCUSSION · 0

No comments yet — be the first to share your approach.