TL;DR: A CPU is built to finish one instruction stream as soon as possible; a GPU is built to finish millions of identical instruction streams per second and does not care how long any one of them waits. That design choice buys an H100 about 3.35 TB/s of memory bandwidth against about 300 GB/s for a two-socket server, and 989 dense bf16 TFLOPS from tensor cores against tens of TFLOPS from AVX-512 or AMX units. Deep learning is almost entirely large, regular, data-parallel matrix arithmetic, which is exactly the workload that trade favors.
How to approach it
Refuse the "thousands of cores" framing and replace it with two mechanisms: how each chip spends its transistors, and how each chip's memory system is built. Give one number for each side of both comparisons, and derive at least one of them so it is not a recitation. Then name what the workload has to look like for the GPU to win, because the interviewer is checking that you know a GPU is not faster at everything. Close with the case where the CPU wins.
A strong answer
A typical situation: a candidate answers "more cores" and the interviewer asks how many cores a 96-core server CPU has against an H100's 132 SMs, which is a ratio of 1.4 and explains none of the 10x to 100x that gets measured. The real answer is two numbers and a design choice.
Start with what each chip is optimizing. A CPU core minimizes latency for a single thread: it spends most of its area on branch prediction, out-of-order windows, speculative execution and a private cache hierarchy, so that one thread almost never stalls. A GPU streaming multiprocessor maximizes throughput across many threads: it spends its area on arithmetic lanes and a 256 KB register file, keeps up to 64 warps resident, and hides a stall by issuing from a different warp on the next cycle. The GPU Execution Model has no branch predictor and no out-of-order window because it does not need one thread to be fast.
The arithmetic gap, derived per clock per core-equivalent:
CPU core with two AVX-512 FMA units:
2 units × 16 fp32 lanes × 2 FLOP (FMA) = 64 FLOP per cycle per core
64 cores × 64 FLOP × 2.5e9 Hz ≈ 10 TFLOPS fp32 per socket, ~ 20 TFLOPS for two sockets
(AMX bf16 tiles raise this by an order of magnitude on recent Xeons, still tens of TFLOPS)
H100 SM, fp32 CUDA cores:
128 lanes × 2 FLOP = 256 FLOP per cycle per SM
132 SMs × 256 × ~1.98e9 Hz ≈ 67 TFLOPS fp32
H100 SM, tensor cores (dense bf16): the 989 TFLOPS datasheet peak implies
989e12 ÷ 132 SMs ÷ ~1.83e9 Hz ≈ 4,096 FLOP per cycle per SM, 16x the fp32 lanes
sanity: the tensor-core figure is 50x the two-socket AVX estimate; the matrix datapath, not
the lane count, is where most of the gap comes from
The memory gap is at least as important, because most deep learning kernels are limited by bytes rather than FLOPs:
server DRAM: 8 channels × 64 bits × 4.8 Gb/s (DDR5-4800) ÷ 8 bits per byte
= 8 × 8 B × 4.8e9 = 307 GB/s per socket
H100 HBM3: 5 stacks × 1,024 bits × 5.23 Gb/s ÷ 8
= 5,120 × 5.23e9 ÷ 8 ≈ 3.35 TB/s
ratio ≈ 11x per socket, ~ 5.5x against two sockets
sanity: HBM gets there with a bus 10x wider (5,120 vs 512 bits) at a similar per-pin rate; the
width is only possible because the stacks sit millimeters from the die on an interposer
Two further things make the GPU's bandwidth usable where a CPU's is not. First, GPU memory access is designed for streaming: a warp reading 32 consecutive floats becomes a handful of 128-byte transactions, and thousands of such warps keep the memory controllers saturated. Second, the GPU can keep enough requests in flight to actually reach 3.35 TB/s, because each SM has dozens of warps waiting on loads simultaneously; a CPU core has a limited number of outstanding misses and relies on prefetchers guessing right.
Now the workload. A transformer forward pass is dominated by GEMMs of shape [tokens × d_model] × [d_model × d_ff] with d_model in the thousands. Every element of the output needs the same multiply-add sequence, the data is contiguous, and the control flow has no data-dependent branches. That is the ideal case for the GPU on both axes: the tensor cores run at their peak on large well-shaped tiles, and the memory system streams weights and activations with perfect coalescing. On the Roofline Model this workload sits far to the right of the ridge, where compute peak is what matters, and the GPU's compute peak is 50x higher.
The reversal condition matters to say out loud. A GPU loses when the work is branchy, pointer-chasing, latency-sensitive or small: a tree search over a linked structure, a request handler that does 2 µs of work per call, a single-row lookup in a hash table. There, the CPU's out-of-order core and low-latency cache win by a wide margin, and the GPU's 600 ns memory latency with no per-thread cache is a liability. Tokenization, request routing, sampling logic that branches per sequence, and the Python that launches kernels all belong on the CPU, and a serving system that has a fast GPU behind a slow CPU is bottlenecked on the CPU more often than people expect.
Decision: for dense tensor arithmetic at batch sizes that fill the tiles, the GPU wins on both bandwidth and FLOPS by an order of magnitude or more, and the cost per useful FLOP follows. The reversal condition: irregular control flow, or working sets so small per request that the tiles never fill. Keep that on the CPU and design the boundary so data crosses PCIe rarely and in large pieces. The GPU Memory Hierarchy is the reason the boundary matters more than the code on either side of it, and nvidia-smi dmon showing high PCIe traffic against low SM activity is what that mistake looks like from the outside.
What interviewers probe next
- "So why do CPUs still get used for inference at all?" Small models, tiny batches and strict cost limits: a 1B model in int8 fits in CPU cache-adjacent DRAM and at batch 1 the GPU is idle 99% of the time anyway; the CPU is cheaper per token there.
- "Why does batch size matter so much on a GPU and less on a CPU?" On a GPU each weight byte read from HBM is amortized over the batch; below a few hundred tokens the tensor cores wait on memory, so throughput scales almost linearly with batch until the ridge.
- "Where does the GPU's power go?" Moving bytes. An H100 at 700 W spends a large share of it on HBM and data movement across the die; a FLOP costs less energy than fetching its operands from DRAM, which is why on-chip reuse is the optimization target.
Common mistakes
- Answering "thousands of cores" and stopping. A GPU lane is not a CPU core, and the count is not where the speed comes from.
- Comparing fp32 CUDA-core TFLOPS with CPU TFLOPS and missing that tensor cores are the 16x factor.
- Forgetting the memory system entirely, which is the half that matters for decode and for most non-GEMM kernels.
- Claiming GPUs are faster at everything, then being unable to explain why the serving framework's scheduler runs on the CPU.
Key takeaways
- Two mechanisms: throughput-oriented execution (lanes and registers instead of speculation) and a streaming memory system (5,120-bit HBM bus, thousands of loads in flight).
- Numbers: ~ 300 GB/s per CPU socket vs 3.35 TB/s; tens of TFLOPS vs 989 dense bf16 TFLOPS, 16x of which is the tensor-core datapath.
- The workload must be large, regular and data-parallel; branchy or latency-bound work belongs on the CPU.
- Most kernels are memory-bound, so the bandwidth ratio matters more often than the FLOPS ratio.