Design an LLM batching system end to end: the queue, the batch, the KV cache and streaming. Give me numbers.

TL;DR: Continuous batching: an iteration-level scheduler admits waiting requests into the running batch every step, runs prefill in chunks alongside decode, and streams each new token to its client through a bounded buffer. On a 70B in fp8 on 8 × H100 the batch is bounded twice: by the TPOT SLO (a step at batch 128 with 2k contexts is about 4.2 ms at the roofline, so 128 fits comfortably under 50 ms even at a third of the roofline) and by KV pages (128 sequences × 0.33 GB = 42 GB of a 505 GB pool). Admission checks free pages against the request's maximum, never preempts a running sequence, and the queue depth is bounded by the wait budget times the service rate.

How to approach it

Fix the model, the hardware and the two SLOs, then say that the batch is the whole design: it sets throughput, TPOT and memory at once. Draw the pipeline from the queue to the client. Derive the batch from the decode roofline and check it against KV capacity, and only then describe the scheduler loop, chunked prefill and streaming. Name the failure at each stage: unbounded queue, KV exhaustion, slow reader.

A strong answer

A typical situation: a 70B dense model in fp8 on one 8 × H100 node, chat traffic with 2,000-token average context and 300-token answers, p95 TTFT 500 ms and p95 TPOT 50 ms.

The batch from the roofline. A decode step reads every weight once and every running sequence's KV once, and produces one token per sequence. Bigger batches amortize the weights; the KV term grows with the batch.

inputs
  weights fp8 = 70.6e9 × 1 B = 70.6 GB
  KV per token, Llama 3.1 70B, fp8 = 2 × 80 layers × 8 KV heads × 128 head dim × 1 B = 163,840 B ≈ 164 KB
  KV per sequence at 2,000 tokens = 164 KB × 2,000 ≈ 0.33 GB
  node bandwidth = 8 × 3.35 TB/s = 26.8 TB/s

step time at batch b = (70.6 GB + b × 0.33 GB) ÷ 26.8 TB/s
  b = 32:  (70.6 + 10.6) ÷ 26.8 = 3.0 ms → 32 ÷ 3.0 ms ≈ 10,600 tok/s
  b = 128: (70.6 + 42.2) ÷ 26.8 = 4.2 ms → 128 ÷ 4.2 ms ≈ 30,000 tok/s
  b = 512: (70.6 + 169) ÷ 26.8 = 8.9 ms → 512 ÷ 8.9 ms ≈ 57,000 tok/s
compute check at b = 512, fp8: 512 × 2 × 70.6e9 ÷ (8 × 1,979e12 × 0.5) ≈ 9.1 ms → past the ridge; compute now co-limits
real engines land at 2 to 3× the roofline step (kernel efficiency, chunked prefill sharing the step)
  b = 128 at 3× = 12.6 ms per step, under the 50 ms SLO with room; b = 512 at 3× = 27 ms, still under
sanity: going from batch 32 to 128 triples throughput for 1.2 ms of step time; that is why batching pays

The KV cap. The pool is what is left after weights and activations.

pool = 8 × 80 GB − 70.6 GB weights − ~40 GB activations and workspace ≈ 530 GB, take 505 GB usable
sequences at 2,000 tokens: 505 ÷ 0.33 ≈ 1,500, so KV is not the cap at this context
at 32k context: 164 KB × 32,768 = 5.4 GB per sequence → 505 ÷ 5.4 ≈ 93 sequences; now KV caps the batch below 128
sanity: a long-context customer changes which bound is binding, and the scheduler must see both

The scheduler loop. Each iteration: finish the running sequences' decode step; free the pages of any that emitted end-of-sequence; admit from the queue while the running batch is under the SLO batch and the request's maximum pages (prompt plus max output tokens) fit in the free list; schedule prefill for new admissions in chunks of about 512 tokens so no single step exceeds the TPOT budget for the sequences already decoding. Chunked Prefill is what keeps a 30,000-token prompt from stalling every other user's token for a second.

Admission and the queue. Continuous Batching without a bounded queue fails the same way any server does: at 5% overload the queue grows by one request per second forever. Depth is bounded by Little's law, wait budget × service rate: at 400 ms of TTFT budget for queueing and 40 admissions per second, that is 16 slots, and the 17th gets a 429 with a retry-after. Admission never preempts running sequences to make room; a preempted sequence either swaps its KV to host memory or recomputes it, and either way every running user sees a TPOT spike.

Streaming. Each admitted request gets a bounded buffer that the decode loop writes to and a connection handler reads from. If a client stops reading, the buffer fills and the request is cancelled and its pages freed; the decode loop never blocks on a socket. The alternative, writing to the socket from the engine thread, means one slow reader stalls a step for 127 other sequences.

The trade-off to commit to is batch size against TPOT. At batch 128 the replica earns about 30,000 tokens per second at the roofline and a comfortable TPOT; pushing to 512 nearly doubles throughput and spends most of the TPOT budget. For interactive chat, stop at the batch where p95 TPOT sits at about 60% of the SLO. The reversal condition: for a batch or agent workload with no human reading the stream, TPOT is irrelevant and the batch grows until KV or compute binds.

Failure modes to name: KV exhaustion from a long-context tenant (cap context per tenant and admit on max pages); a prefill-heavy burst starving decode (chunk size and a cap on prefill tokens per step); a slow client (bounded buffer, cancel); a retry storm after rejections (clients back off with jitter).

What interviewers probe next

• "Why not admit on the current prompt length instead of prompt plus max output?" Because the sequence grows a page at a time and a pool that admits on current length preempts when it runs out; reserving the maximum wastes pages but never preempts, and a per-tenant max output keeps the waste bounded.
• "Where does the 50 ms TPOT go if the batch is 4 ms at the roofline?" Kernel efficiency, chunked prefill sharing the step, the attention kernel at long contexts, and the tail of the step-time distribution; the SLO is on p95, and the p95 step is well above the mean.
• "How would you know the batch is too big?" p95 TPOT rising toward the SLO with KV utilization flat; if instead KV utilization is pinned and preemptions are nonzero, the pool is the bound.

Common mistakes

• Picking a batch size from a benchmark without deriving it from the SLO and the roofline.
• Treating the KV pool as a cache to be evicted from, rather than the admission currency.
• Writing tokens to the socket from the engine loop.
• Forgetting that prefill and decode share the step and quoting decode-only throughput.

Key takeaways

• Step time = (weights + batch × KV per sequence) ÷ bandwidth; on a 70B fp8 node batch 128 is about 4 ms at the roofline.
• KV per token for a 70B is 164 KB in fp8; at 32k context a sequence is 5.4 GB and KV becomes the binding cap.
• Admit on free pages against the request's maximum; never preempt running sequences; bound the queue by wait budget × service rate.
• Streaming through bounded buffers so a slow reader costs one request, not the batch.