TL;DR: Gateway (auth, quotas, validation, streaming), router (model to pool, replica by prefix hit and load), engine scheduler (continuous batching, chunked prefill, admission), replicas (a model copy on a tensor-parallel group), KV tier (paged pool on the GPU, host offload above it), autoscaler (replica count from queue depth and KV pressure, with a warm pool), observability (TTFT, TPOT, goodput, KV utilization). Sizing goes demand in tokens per second divided by goodput per replica, plus headroom: 2,000 concurrent chat users on a 70B model is about 9 eight-GPU replicas.
How to approach it
Ask what the product is (chat, completions, batch), the model size, the peak concurrency and the two latency SLOs, then say you will draw the standard shape and size it. Draw left to right from the client's point of view, one job per box, and say the failure each box owns as you draw it. Convert users to tokens per second before any GPU count. Offer the router and the engine scheduler as the places to go deeper, since that is where the interviewer wants to spend the time.
A strong answer
A typical situation: the screen asks for the platform and gets a component list, because the candidate is naming things rather than saying what each box is for and how it fails. Seven boxes, one job and one failure each, is the shape that survives the follow-ups.
A typical opening: a chat product with 2,000 concurrent users at peak on a 70B dense model, p95 TTFT 500 ms and p95 TPOT 50 ms, hosted on 8 × H100 nodes. The Inference Platform Architecture has seven boxes, and the walkthrough follows a request through them.
Gateway. Terminates TLS, checks the API key, applies per-tenant limits in tokens per minute and concurrent streams, validates that the model exists and the context fits, and proxies the token stream back. It is stateless. Its failure: a limiter kept per instance instead of shared, which lets a tenant exceed its quota by the number of gateway instances.
Router. Maps the model name to a pool, then picks a replica. Round-robin is the naive answer; the answer that matters is prefix-aware, because a multi-turn conversation whose earlier turns are cached on replica 7 should go back to replica 7, turning a 3,000-token prefill into a few hundred. Load (queue depth, KV utilization) breaks ties so no replica saturates. Its failure: a replica restarts, its cache is empty, and every conversation pinned to it pays a full prefill at once.
Engine scheduler. Inside vLLM or SGLang: each iteration admits waiting requests into the running batch, mixes prefill chunks with decode steps so TTFT and TPOT both stay bounded, and refuses admission when the KV pool has no free pages. Its failure: admitting past the pool and preempting running sequences, which spikes TPOT for every user on that replica.
Replicas. One model copy each. A 70B in bf16 is 70.6e9 × 2 B = 141 GB of weights, more than one 80 GB card, so a replica is a tensor-parallel group of 8 GPUs on one node. An 8B fits on one GPU.
KV tier. The paged pool on each replica's GPUs is the first tier. Host memory offload lets a paused session come back without a re-prefill, and a remote pool over RDMA lets prefixes be reused across replicas so routing affinity can be looser.
Autoscaler. Replica count from queue depth, KV pressure and TTFT headroom, with the cold start counted: a 70B replica takes a minute or more to load weights and warm up, so the scaler acts on a leading indicator and keeps a warm pool.
Observability. Per request: TTFT, TPOT, tokens in and out, cache hit, replica. Per replica: batch size, KV utilization, queue depth. Per pool: goodput against the SLO.
The sizing chain, stated so the interviewer can change any input:
demand
fraction of users mid-generation ≈ answer time ÷ (answer + read-and-type time)
answer ≈ 300 tokens × 50 ms = 15 s; read and type ≈ 45 s → 15 ÷ 60 = 25%
decode demand = 2,000 × 0.25 × 20 tok/s = 10,000 tok/s
prefill demand = 2,000 users × (1 request ÷ 60 s) × 500 uncached tokens ≈ 17,000 tok/s
supply per replica (8 × H100, TP8, bf16)
decode ceiling from bandwidth: bytes per step = 141 GB + 64 seqs × (320 KB × 2,000 ctx = 0.64 GB) = 182 GB
step = 182 GB ÷ (8 × 3.35 TB/s = 26.8 TB/s) = 6.8 ms → 64 ÷ 6.8 ms ≈ 9,400 tok/s ceiling
measured goodput at p95 TPOT 50 ms, with prefill interleaved and attention kernels below roofline: take 2,000 tok/s
replicas
decode: 10,000 ÷ 2,000 = 5 replicas if decode had the GPUs alone
prefill takes about 30% of each replica: 5 ÷ 0.7 ≈ 7.2 → 8
headroom for one failed replica: 9 replicas = 72 H100s
cost: 72 × $2.50/h = $180/h at peak ≈ $130k/month if held at peak size all day
sanity: 10,000 users at the same ratios is about 45 replicas, which matches the reference derivation
The decision inside the shape is where prefix affinity lives. Replica-local caches with sticky routing are the right call at this scale, because a remote KV pool costs a network hop per hit and an operations burden the team does not need yet. The reversal condition: when replica restarts or rollouts produce visible TTFT spikes because conversations lose their cache, or when the pool grows past a few dozen replicas and affinity fragments load, a shared prefix tier becomes worth its cost. Inference Platform Architecture is the same seven boxes at rest, and Request Routing and Load Balancing for LLMs is the box in front of them.
Failure modes to name unprompted: the KV pool fills and admission must queue rather than preempt; a rollout moves the hash ring and prefill demand jumps as caches miss; a throttled GPU makes one replica slow while least-loaded routing hides it; a traffic step arrives faster than a cold start and the gateway must shed with a retry-after.
What interviewers probe next
- "Traffic doubles in five minutes; what happens?" The autoscaler is a minute behind, the warm pool absorbs the first step, and beyond that the gateway rejects with a retry-after rather than letting queues grow.
- "Why tokens per second and not requests per second?" A 500-token prompt and a 30,000-token prompt are the same request and 60× the work; sizing in requests is off by whatever the length distribution hides.
- "Where does the batch size come from?" From the TPOT SLO: the batch at which p95 TPOT crosses 50 ms, measured per model and hardware, is the replica's goodput.
Common mistakes
- Drawing a load balancer and a "GPU workers" box with nothing inside; the engine scheduler and the KV pool are the design.
- Sizing from requests per second, or not sizing at all.
- Forgetting that prefill shares the replica with decode and dividing only by decode goodput.
- Describing the autoscaler as CPU-utilization-driven; GPU utilization is a lagging signal, and queue depth plus KV pressure lead.
Key takeaways
- Seven boxes, each with a job and a failure: gateway, router, engine scheduler, replicas, KV tier, autoscaler, observability.
- 70B bf16 = 141 GB, so a replica is 8 GPUs; KV per token on a 70B is 320 KB in bf16.
- Demand in tokens per second ÷ goodput per replica, divided again by decode's share, plus headroom: 2,000 users ≈ 9 replicas.
- Prefix affinity in the router and admission in the scheduler are the two deep dives worth the time.