hardNewFireworks

Design the fastest serving path for a 70B model with a 200 ms TTFT target at production scale.

Two hundred milliseconds is about four thousand prompt tokens of prefill on a full node, before any queueing. Every design choice is about not spending that budget twice.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

Two hundred milliseconds is about four thousand prompt tokens of prefill on a full node, before any queueing. Every design choice is about not spending that budget twice.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🧮 Open Weights & Serving Engines

vLLM Server Arguments That MatterA vLLM deployment is mostly decided by a dozen flags, and the ones that matter fall into four groups: how the model is split across GPUs, how memory is divided between weights and cache, how requests are batched, and which specialized backends the model needs. Getting the first two wrong produces an engine that will not start or that runs out of memory under load. Getting the third wrong produces an engine that starts, serves, and misses its latency target by a wide margin.

Advanced

🚀 Inference & Serving🔒 Premium

Chunked PrefillA long prompt's prefill can occupy a GPU for hundreds of milliseconds, and every sequence mid-decode on that GPU waits for it. Chunked prefill splits the prompt into fixed token budgets and interleaves each chunk with a decode step, so decode latency stays flat at the cost of a slower first token for the long prompt. The chunk budget is a knob between TTFT and TPOT, and the interview question is how you would set it.

Advanced

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodePrefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.

Core

📐 AI Systems DesignSign in

Designing for Latency SLOsA latency objective is met or missed by the sum of a chain of delays, and the way to design for it is to write the chain down with a number on every link, find the links that dominate at the tail, and attack those. For an LLM request the chain is network, gateway, router, queue, prefill, then the decode loop, and the tail is shaped by queueing and by the size of the batch the request lands in. This page decomposes a 500 ms time-to-first-token budget link by link, derives how queueing turns a comfortable median into a broken p99, and gives the design moves (admission control, chunked prefill, priority lanes, hedging) that hold it.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this tracknvidia-smi shows 30 percent utilization on our serving fleet. Is that a problem, and what would you look at instead?Next in this trackWhat changes when you serve a vision-language model instead of a text-only one?Next in this trackWhen does offloading the KV cache to CPU memory or NVMe beat recomputing it?

FEDITOR'S NOTE

Scored on turning the SLO into a token budget with the prefill formula, on layering the levers in order of payoff (TP8, fp8, prefix cache, chunked prefill, disaggregation), and on stating what the design gives up to hit the number.

DISCUSSION · 0

No comments yet — be the first to share your approach.