TL;DR: Two tiers with one budget. Retrieval: embed (about 10 ms), search 8 document shards in parallel on a compressed index (about 30 ms), rerank the top 50 with a cross-encoder (about 60 ms). Inference: a 70B in fp8 on 8 × H100 nodes fed 3,000-token retrieval-augmented prompts, which are prefill-dominated and mostly uncached, so the LLM fleet is sized by prefill FLOPs. At 10k QPS with 40% of queries answered from an answer cache and 30% of the rest needing the LLM, the LLM tier is about 60 nodes with headroom; without the cache it is closer to 100. The p95 end-to-end TTFT target of 1.5 s is met with about 900 ms to spare for queueing.
How to approach it
Ask what fraction of queries need a generated answer, how fresh the index must be, the corpus size, and the end-to-end latency target. Say the budget has to be decomposed per stage before any tier is sized. Draw the request path, then size retrieval in shard-queries per second and reranking in pairs per second, and size inference in prefill tokens per second. Put the cache in the chain as a number, not a footnote. Close with the SLOs per stage and the failure modes.
A strong answer
A typical situation: a search product over 100 million documents, 10,000 queries per second at peak, a generated answer on top of the results for queries that look like questions, p95 TTFT for the answer under 1.5 s and p95 for the result list under 300 ms.
The budget. Every stage gets a p95 allowance, and the sum is under the target with slack for queueing: embed 10 ms, search 30 ms, rerank 60 ms, LLM prefill 70 ms of compute plus queue, streaming setup 10 ms. That is about 180 ms of work against a 1,500 ms target, so the design has about 1,300 ms to spend on queues, retries and the tail of the slowest shard, which is where Designing for Latency SLOs pays off: the fan-out to 8 shards means the p95 of the query is set by the p99 of a shard.
The retrieval tier.
index
100 M documents × 1,024-dim embeddings; product-quantized to 64 B per vector = 6.4 GB, plus graph ≈ 12 GB
8 shards by document hash → 1.5 GB per shard, in memory, one CPU node each
throughput
every query fans out to all 8 shards: 10,000 QPS × 8 = 80,000 shard-queries/s
one shard replica serves about 2,000 QPS at p99 under 15 ms (an assumption to measure)
replicas per shard = 10,000 ÷ 2,000 = 5; +1 for failure → 6 per shard → 48 search nodes
reranker
pairs per second = 10,000 × 50 = 500,000; each pair ≈ 300 tokens → 150 M tokens/s
a 300M-parameter cross-encoder: 2 × 3e8 × 1.5e8 = 9e16 FLOP/s
per H100 at fp8, MFU 0.5: 1,979e12 × 0.5 ≈ 1e15 → 90 GPUs; +20% → 108, so 14 nodes of 8
sanity: the reranker costs more GPUs than the embedder by two orders of magnitude; rerank fewer
candidates (top 20) and the tier shrinks 2.5×
The inference tier. A retrieval-augmented prompt is 3,000 tokens of freshly retrieved passages, so only the system prompt (about 500 tokens) is a prefix hit; the other 2,500 are uncached prefill every time. The Inference Platform Architecture is the reference shape, but here the sizing is dominated by the prefill column.
which queries reach the LLM
answer cache on the normalized query with near-duplicate matching: 40% hit rate (assumed; measure it)
of the 6,000 misses, 30% are question-shaped and get a generated answer → 1,800 req/s
demand
prefill: 1,800 × 2,500 uncached = 4.5 M tok/s; decode: 1,800 × 200 output = 360,000 tok/s
supply per 8 × H100 node, 70B fp8
prefill at MFU 0.4: 8 × 1,979e12 × 0.4 ÷ (2 × 70.6e9) ≈ 45,000 tok/s
decode, conservative: 25,000 tok/s
nodes
prefill: 4.5 M ÷ 45,000 = 100; decode: 360,000 ÷ 25,000 ≈ 15
in one pool with chunked prefill the prefill column dominates: ~115 nodes; with disaggregation, 100 + 15
headroom 20% → about 140 nodes ≈ 1,100 H100s
without the answer cache: 3,000 req/s → prefill 7.5 M tok/s → 167 prefill nodes → ~220 nodes total
TTFT: 2,500 tokens × 2 × 70.6e9 ÷ (8 × 1,979e12 × 0.4) ≈ 56 ms of compute; the rest of the 1.5 s is queue
cost: 140 × 8 × $2.50/h = $2,800/h for the LLM tier alone
sanity: 1,800 generated answers per second on 1,100 GPUs is about 1.6 answers per GPU-second; the prompt
length is the reason, and shortening passages from 3,000 to 1,500 tokens halves the tier
The trade-off to commit to: an answer cache in front of the LLM, keyed on normalized query with near-duplicate matching, with a TTL tied to index freshness. It removes 40% of the most expensive work and it is what makes the tier 140 nodes instead of 220. The reversal condition: if the corpus updates faster than the TTL can follow (news, prices) or the queries are long-tail with a measured hit rate under 10%, the cache is dead weight and the money goes into shorter prompts and a smaller answer model instead.
Failure modes to name: a slow shard makes every query slow (hedge the fan-out after the p90, and eject the replica); the reranker saturates and the queue in front of it eats the budget (bound it and fall back to the ANN order); a burst of question-shaped queries hits the LLM tier's prefill cap (shed to "results only" for the excess, with the answer arriving asynchronously); index rebuilds during peak (swap shard replicas one at a time); a cache poisoned by a bad model rollout (version the cache key with the model).
What interviewers probe next
- "Why is the LLM tier prefill-bound when chat platforms are decode-bound?" Chat has 80% prefix hits and 300-token answers; RAG has fresh passages every time and 200-token answers, so the ratio of uncached prefill to decode is 12:1 here against about 1:1 there.
- "Where do you put the budget if the target drops to 800 ms?" Cut rerank candidates to 20, cap passages at 1,500 tokens, and reserve prefill capacity at 60% utilization so the queue term shrinks; the arithmetic shows the work is only 180 ms.
- "How do you keep the index fresh?" Incremental upserts into a delta segment per shard, merged into the main index on a schedule, with the answer cache TTL matched to the merge cadence.
Common mistakes
- Sizing the LLM tier by requests per second with a chat-like prefix hit rate.
- Ignoring the reranker's cost, which can exceed the embedder's by 100× and rival the answer model's.
- A single end-to-end SLO with no per-stage budget, so nobody knows which stage to fix when it slips.
- Treating the answer cache as an afterthought instead of a line in the sizing chain.
Key takeaways
- Decompose the budget: embed 10, search 30, rerank 60, prefill 70 ms of work; the remainder is queue and tail.
- RAG prompts are prefill-bound: 1,800 req/s × 2,500 uncached tokens = 4.5 M tok/s → 100 prefill nodes on a 70B.
- Reranking 500k pairs/s on a 300M cross-encoder is about 90 H100s; candidates count is the lever.
- The answer cache is a fleet-sizing input: 40% hits is the difference between 140 and 220 nodes.