Anthropic LLM Inference & Serving interview questions

One forward pass reads every weight. Whether that read is the bottleneck depends on how many tokens ride along with it, and the answer is different for the two halves of a request.

One 20k-token prompt can freeze every active stream for most of a second. The fix slices it into per-step budgets, and the budget number is a trade between two SLOs you can compute.

A small model guesses four tokens and the big one checks them in a single step. That converts idle bandwidth into tokens, and the arithmetic tells you exactly which batch size stops it working.

Prefill and decode fight over the same GPU and each ruins the other's latency. Putting them on separate pools ends the fight, at the price of shipping every request's cache across the network. The break-even is a number you can derive.

The same 2,000 tokens are prefilled a thousand times an hour. Caching their KV by content turns that into one prefill and a table lookup, and a radix tree is what makes multi-turn and branching agents share it too.

Sequential is fifty seconds; one batch is four. In between are the questions the interviewer is holding: how you pack ragged inputs, what a batch window costs, and where the memory stops you.

A flat median with a broken tail means one in a hundred requests is hitting something the others do not. There are five usual suspects, and the order you rule them out in is the answer.

Least-loaded routing sends a conversation's tenth turn to a replica that has never seen it, and the whole history prefills again. Affinity fixes that and creates hot spots. The router that does both is a scoring function with two terms.

At 128k tokens one request's cache is 43 gigabytes and its prefill is measured in seconds. Capacity breaks first, then TTFT, then the scheduler. Each has a fix, and the numbers say which fix you need at which length.

Temperature zero removes the sampling randomness and leaves the floating-point kind. The batch your request lands in changes the reduction order, the logits move in the last bits, and a near-tie flips a token. The fix has a cost.

Ten thousand requests a second is a fleet of hundreds of nodes, a router that has to know what every replica is caching, and an SLO pair that decides the batch on every one of them. Here is the sizing chain, node by node.

Every token a model has seen leaves a key and a value in every layer. Multiply that out for a 70B model and you will see why memory, not compute, caps how many users a GPU can hold.

A batch of eight requests finishes when the longest one does, and seven slots compute padding until then. The fix is to make the scheduling unit one decode step, and the throughput math shows why that was worth several times the hardware.

Before vLLM, engines reserved the maximum context for every request up front and wasted most of it. The fix borrowed virtual memory from the operating system, and the numbers show how much capacity it returned.

Tokens per second flatters a system that is failing its users. The metrics that matter are two latencies at a percentile and the fraction of requests that meet both.

Weights take a fixed slice, the KV cache takes the rest, and the rest divided by context is your user count. Work it in bf16 and fp8, then find out why the answer also depends on how you split the model.

Same model size, three attention layouts, a 30x spread in cache per token. The formulas are short, and they explain why DeepSeek can hold four times the context of a dense 70B on the same cards.