AI Infrastructure System Design: the practice test
Design an inference platform at 10k requests per second, a 10,000-GPU training cluster, a job scheduler with preemption and checkpointing, a serverless GPU runtime with sub-second cold starts, a multi-tenant fine-tuning service, an eval pipeline. The whiteboard round at OpenAI, Anthropic, Baseten and Together. This test drills exactly that: 10 easy, 10 medium and 10 hard questions, every one explained, every explanation linking into the worked material.
Sample questions, answered
The gateway authenticates, rate-limits and validates; the router picks a replica by prefix locality and load; the engine's scheduler batches and admits requests; replicas hold model copies across one or more GPUs; the KV tier is the paged pool on GPU with optional offload; the autoscaler sizes the pool with cold starts in mind; observability tracks TTFT, TPOT, goodput and KV utilization. A design answer names each box's job and its failure mode, then sizes the pool with a stated chain of arithmetic.
A 300-token completion and a 30,000-token summary are one connection each, but differ by two orders of magnitude in prefill work and KV footprint; and a replica that already holds a conversation's prefix answers in 12 ms of prefill where a cold one needs 200 ms. A cost-aware router scores each replica as expected prefill after cache hits plus expected queue wait, reading pending tokens, running sequences and KV occupancy from the engines, and picks the minimum; near saturation the queue term overrides affinity.
Go deeper than the quiz
A practice test measures recall. The material it draws from teaches the reasoning: