OpenAI AI Infrastructure System Design interview questions

Seven boxes between an API call and a GPU, each with one job and one way to fail. The walkthrough a screen expects in the first ten minutes, with the sizing chain from 2,000 concurrent users to a replica count so the drawing has numbers on it.

A GPU credit system is an accounting ledger with a scheduler attached: who may spend, how fast, what happens when the account is empty and what happens when everyone spends at once. The data model, the debit path, the fairness rule and the arithmetic that keeps 4,096 GPUs busy without starving a team.

A hundred thousand jobs is a queue you cannot scan per tick, a preemption policy that has to know when each victim last checkpointed, and a state machine with one transition that most designs get wrong. The data model, the tick loop, the checkpoint-aware eviction and the arithmetic behind each.

Two thousand evals against every checkpoint is ten million prompts per run and the difference between a four-hour and a two-day turnaround. The pipeline as a batch inference job with a content-addressed cache, the sharding that keeps GPUs busy, and the reproducibility rules that let a regression be believed.

A request can cost 50 tokens or 50,000, so a request limit protects nothing. The two buckets per tenant per model, the reservation-then-settle scheme for output tokens you cannot count in advance, the distributed counter fast enough for the gateway, and how the limits map onto the fleet's real capacity.

Two regions at 60% is not one region with a spare: the question is what happens in the 30 seconds after a region drops. Capacity sized for N-1 with the arithmetic, latency routing with a residency override, failover that does not stampede, weights warm everywhere in advance, and the state that must not cross a border.

The playground is the API's most public client and its most abused one: free, barely identified, streaming. The transport and its cost per connection, session state that survives a refresh with no GPU holding anything, quotas per user and per fleet, and the abuse controls that stop the free tier becoming a free API.

GPU utilization at 100% tells you nothing, and a trace per token would cost more than the tokens. The span layout for one request, the GPU counters that read headroom, the cardinality budget that keeps the metrics store alive, and cost attribution that charges tenants for the batch share they used.

An agent thinks for a second, runs a tool for a minute, and repeats a hundred times over an hour. The session state machine and where each piece lives, the sandbox per session with its cold-start budget, the KV cache that should survive the tool call, and checkpoints that resume a step rather than an hour.

A video request is not a stream of tokens: it is 40 denoising passes over a latent the size of a small film, and the user waits a minute for the whole thing. The memory and FLOP arithmetic per request, why batching works differently, the queue and progress design for minute-long jobs, and where the cost goes.

A research cluster is shared by teams whose managers bought it and by researchers who want it now. A quota tree with guarantees and borrowing, a fair-share formula that decays so last week's usage does not cost you today, preemption that is cheap because checkpoints are, and the incentives that make hoarding lose.

Billing is the one pipeline where a dropped event is money and a duplicated one is a support ticket. Where tokens are counted and by whom, the usage event and its idempotency key, the aggregation that closes a month, the reconciliation that proves the ledger against the engines, and the six ways it goes wrong.

A trillion parameters with 32 billion active per token: the weights need a rack, the compute needs a fraction of it, and the all-to-all between them decides everything. The memory plan across 72 GPUs, the decode step from bandwidth and two all-to-alls, the compressed attention cache, and what imbalance costs.

The first question is whether 1T is total or active, because storage follows one and compute follows the other. The state budget, the layout that falls out of it, why expert parallelism belongs inside NVLink, and the two failure modes a dense-model plan does not have: router imbalance and all-to-all congestion.

Five minutes of numbers, seven of shape, ten of arithmetic, sixteen of depth, five of failure modes. The minute plan that keeps a serving-platform design inside 45 minutes, with the worked opening for a coding assistant and the mistakes that end the round early.

From an admission queue to a streamed token, the batching system that decides how many tokens per second a replica earns and what its p95 TPOT is. The decode arithmetic that sets the batch, the KV budget that caps it, and the streaming path that must never stall the engine.

Ten thousand queries a second through an embedder, a sharded vector index, a reranker and a 70B answer model. The latency budget per stage, the two fleets sized from tokens and pairs rather than queries, and the cache that decides whether the LLM tier is 60 nodes or 200.

Ten thousand GPUs is 1,280 nodes, ten pods, about 700 switches, a filesystem that swallows a 5 TB checkpoint in under a minute and a 14 MW feed. The bill of materials with a reason on every line, from the FLOPs the model needs to the spare pool the failure rate demands.

A serverless GPU is a promise to run a customer's model within seconds of a request while charging only for the seconds it runs. The cold-start chain from image pull to first token, the snapshot that collapses it, the warm pool sized by Little's law, and the isolation and packing decisions that set the margin.

Fifteen trillion tokens starts as a few petabytes of raw text and ends as 30 TB of shards a training job reads at 1.7 million tokens per second. The stages, the bytes at each boundary, the throughput each one needs to finish in two weeks, and the two stages where the pipeline actually spends its time.