expertNewNVIDIAAnthropicOpenAI

If you could change one thing about GPU architecture for LLMs, what would it be, and what argues against it?

The workload asks for bytes and the chip delivers FLOPs: bytes per FLOP halved from A100 to H100 and held level on B200. A defensible thesis is more bandwidth and capacity per FLOP, the shoreline and power arithmetic for why it has not happened, and the counterargument that software already routes around the wall.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🧩 GPU & Accelerator Architecture

GPU Memory HierarchyA GPU has four places a byte can live, and they differ by a thousandfold in bandwidth: registers, shared memory on the SM, a chip-wide L2, and HBM off-chip. Almost every kernel optimization is a decision about which level a value is read from and how many times. Knowing the sizes and bandwidths for an H100 cold is what lets you say why a kernel is slow before you profile it.

Foundational

🚀 Inference & Serving

The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

ECC, Row Remapping and Memory ErrorsHBM stacks flip bits, and the difference between a fleet that shrugs and one that loses a training step to corruption is error-correcting codes plus the machinery that retires bad memory before it produces a double-bit error. A single-bit error is corrected silently and counted; a double-bit error is detected, kills the process, and on Ampere and later triggers the row remapper to swap the failing row for a spare at the next reset. This page explains the codes, the remapper's states, how to read the counters as a prediction of failure, and the RMA rules a fleet applies.

Foundational

🧩 GPU & Accelerator Architecture

GPU Execution ModelA GPU hides memory latency with parallelism instead of caches: thousands of threads in flight, scheduled in warps of 32, pinned to streaming multiprocessors that switch between warps for free whenever one stalls. Every performance conversation in an AI infra loop, from occupancy to why decode is slow, rests on this one mechanism.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackOne node in your cluster shows half the expected NVLink bandwidth in nccl-tests. Walk me through isolating it.Closest to this topicExplain what HBM is and why memory bandwidth, not compute, is the wall for LLM inference.Closest to this topicWhy are GPUs so much faster than CPUs for deep learning? Be specific about what the silicon is doing differently.

FEDITOR'S NOTE

Scored on a thesis that follows from measured ratios, an honest account of the physical reasons the obvious fix is hard, and a counterargument the candidate takes seriously rather than knocks down.

DISCUSSION · 0

No comments yet — be the first to share your approach.