Anyscale AI Infrastructure System Design interview questions

A hundred thousand jobs is a queue you cannot scan per tick, a preemption policy that has to know when each victim last checkpointed, and a state machine with one transition that most designs get wrong. The data model, the tick loop, the checkpoint-aware eviction and the arithmetic behind each.

A replica per variant is 100 copies of a 140 GB base and 800 GPUs mostly idle; multi-LoRA holds the base once and hundreds of adapters in a few gigabytes. The memory arithmetic, the throughput cost of mixed batches, the routing that keeps hot adapters resident, and the two cases where a variant earns its own replica.

Seven boxes between an API call and a GPU, each with one job and one way to fail. The walkthrough a screen expects in the first ten minutes, with the sizing chain from 2,000 concurrent users to a replica count so the drawing has numbers on it.

Five minutes of numbers, seven of shape, ten of arithmetic, sixteen of depth, five of failure modes. The minute plan that keeps a serving-platform design inside 45 minutes, with the worked opening for a coding assistant and the mistakes that end the round early.

From an admission queue to a streamed token, the batching system that decides how many tokens per second a replica earns and what its p95 TPOT is. The decode arithmetic that sets the batch, the KV budget that caps it, and the streaming path that must never stall the engine.

A GPU credit system is an accounting ledger with a scheduler attached: who may spend, how fast, what happens when the account is empty and what happens when everyone spends at once. The data model, the debit path, the fairness rule and the arithmetic that keeps 4,096 GPUs busy without starving a team.

Ten thousand queries a second through an embedder, a sharded vector index, a reranker and a 70B answer model. The latency budget per stage, the two fleets sized from tokens and pairs rather than queries, and the cache that decides whether the LLM tier is 60 nodes or 200.

Ten thousand GPUs is 1,280 nodes, ten pods, about 700 switches, a filesystem that swallows a 5 TB checkpoint in under a minute and a 14 MW feed. The bill of materials with a reason on every line, from the FLOPs the model needs to the spare pool the failure rate demands.

A serverless GPU is a promise to run a customer's model within seconds of a request while charging only for the seconds it runs. The cold-start chain from image pull to first token, the snapshot that collapses it, the warm pool sized by Little's law, and the isolation and packing decisions that set the margin.

Two thousand evals against every checkpoint is ten million prompts per run and the difference between a four-hour and a two-day turnaround. The pipeline as a batch inference job with a content-addressed cache, the sharding that keeps GPUs busy, and the reproducibility rules that let a regression be believed.

Fifteen trillion tokens starts as a few petabytes of raw text and ends as 30 TB of shards a training job reads at 1.7 million tokens per second. The stages, the bytes at each boundary, the throughput each one needs to finish in two weeks, and the two stages where the pipeline actually spends its time.

A replica takes 70 seconds to become useful and the burst arrives in 60, so the autoscaler cannot win the first minute; the warm pool has to. The leading signal, the cooldowns derived from the cold start, the warm-pool size from Little's law and the burst profile, and the cost of every second of cold start left in.

A 70B model's KV cache is 320 KB per token, and a 30k-token conversation is 10 GB that GPU memory cannot keep between turns. The three tiers with their bandwidths, the break-even where reloading a prefix beats recomputing it, the eviction and lookup design, and the traffic shape where the tier is worth its complexity.

A 141 GB artifact is not a Docker image, and shipping it like one costs an hour per node and a registry that melts under a rollout. Content-addressed chunks, a manifest per version, a fan-out that turns one origin into a thousand nodes in minutes, integrity that fails closed, and promotion tied to evaluation.

A request can cost 50 tokens or 50,000, so a request limit protects nothing. The two buckets per tenant per model, the reservation-then-settle scheme for output tokens you cannot count in advance, the distributed counter fast enough for the gateway, and how the limits map onto the fleet's real capacity.

Two regions at 60% is not one region with a spare: the question is what happens in the 30 seconds after a region drops. Capacity sized for N-1 with the arithmetic, latency routing with a residency override, failover that does not stampede, weights warm everywhere in advance, and the state that must not cross a border.

The playground is the API's most public client and its most abused one: free, barely identified, streaming. The transport and its cost per connection, session state that survives a refresh with no GPU holding anything, quotas per user and per fleet, and the abuse controls that stop the free tier becoming a free API.