Implement the verification step of speculative decoding, including the rollback of the KV cache after a rejection.

The target runs one forward pass over all k drafted tokens, compares its distribution against the draft's at each position, and keeps the longest prefix that survives a coin flip. The acceptance rule that makes the output exactly a target sample, the resampling at the first rejection, and the one-integer rollback.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🚀 Inference & Serving🔒 Premium

PagedAttentionPagedAttention stores the KV cache in fixed-size blocks scattered across HBM and maps each sequence's logical positions to physical blocks through a block table, the same trick an operating system uses for virtual memory. It removes the reservation and fragmentation waste of contiguous allocation, lets blocks be shared between sequences, and is why an engine can decide admission by counting free blocks.

Advanced

🧭 Ownership & Judgment🔒 Premium

Migrations and DeprecationsEvery infrastructure career contains a migration nobody wanted: the scheduler swap, the driver upgrade across a live fleet, the storage move while training runs are in flight, the deprecation of the launcher every team's scripts depend on. The behavioral round asks about one because it tests the skills that matter most and show least on a résumé: sequencing under risk, keeping a rollback real, moving people who have no reason to move, and knowing when to stop. This page gives the shape of a migration story that lands, two worked narratives from GPU fleet work, and the answers that sound like leadership and fail.

Advanced

🚀 Inference & Serving🔒 Premium

Speculative DecodingDecode is memory-bound: each step reads every weight to produce one token. Speculative decoding has a cheap draft propose several tokens, then verifies them all in one forward pass of the big model, so one weight read yields several tokens with output distribution unchanged. It wins 2x to 3x at small batch, breaks even near the ridge point where the GPU is already compute-bound, and lives or dies on the acceptance rate, which is what interviewers ask you to reason about.

Foundational

🚀 Inference & Serving

The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow do you overlap host-to-device transfers with compute, and what has to be true before the overlap actually happens?Next in this trackWhy do tensor cores accumulate in fp32 when the inputs are bf16, and why do optimizers keep fp32 master weights?Next in this trackIn an int8 or fp8 GEMM, what exactly gets scaled, where does the dequantization happen, and what breaks if you get it wrong?

FEDITOR'S NOTE

Scored on stating the accept-reject rule correctly including the residual distribution, on knowing the output distribution is exact rather than approximate, and on the rollback being a length update rather than a memory operation.

DISCUSSION · 0

No comments yet — be the first to share your approach.