hardNewFireworksAnthropic

Design an LLM service for a 200 ms time-to-first-token SLO at p99. Decompose the budget and say what you would give up.

200 milliseconds at p99 is a budget, and prefill alone spends it on a long prompt. The decomposition into network, queue, routing, prefill and first token, the prompt length the budget allows, chunked prefill in the tail, admission control that rejects what it cannot serve in time, and the utilization the SLO costs.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core

📐 AI Systems DesignSign in

Designing for Latency SLOsA latency objective is met or missed by the sum of a chain of delays, and the way to design for it is to write the chain down with a number on every link, find the links that dominate at the tail, and attack those. For an LLM request the chain is network, gateway, router, queue, prefill, then the decode loop, and the tail is shaped by queueing and by the size of the batch the request lands in. This page decomposes a 500 ms time-to-first-token budget link by link, derives how queueing turns a comfortable median into a broken p99, and gives the design moves (admission control, chunked prefill, priority lanes, hedging) that hold it.

Advanced

📐 AI Systems Design🔒 Premium

Capacity and BackpressureA system that accepts more work than it can finish does not degrade gracefully; it degrades completely, because every request it queues makes every other request slower until all of them time out. The defence is backpressure: bounded queues at every stage, admission control that rejects early when the expected wait exceeds the budget, load shedding by priority, and clients that back off. For LLM serving the stages are the gateway, the router, the engine's queue and its KV pool. This page works the arithmetic of why unbounded queues kill p99, designs the bounds per stage, and the client contract that keeps it stable under overload.

Advanced

🚀 Inference & Serving🔒 Premium

Chunked PrefillA long prompt's prefill can occupy a GPU for hundreds of milliseconds, and every sequence mid-decode on that GPU waits for it. Chunked prefill splits the prompt into fixed token budgets and interleaves each chunk with a decode step, so decode latency stays flat at the cost of a slower first token for the long prompt. The chunk budget is a knob between TTFT and TPOT, and the interview question is how you would set it.

Foundational

🚀 Inference & Serving

Prefill vs DecodeAn LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the runtime for long-running agents: sessions, tool sandboxes, KV that lives for hours, and checkpoints.Next in this trackDesign serving for a video generation model: diffusion steps, batching, memory, and a latency profile unlike an LLM's.Next in this trackDesign speculative decoding into a production serving fleet: draft placement, acceptance monitoring, and the batch regime where it pays.

FEDITOR'S NOTE

Scored on decomposing the 200 ms into stages with numbers, on computing the prompt length the budget permits, on separating traffic classes so long prompts do not set the tail, and on admission control and the utilization cost of a tight p99.

DISCUSSION · 0

No comments yet — be the first to share your approach.