AI Infra Interviews logo
← Napkin Math, Cost & Capacity / 11
mediumNewAnthropicFireworks

How long does prefill take for an 8k-token prompt on a 70B model?

Prefill is the compute-bound half of serving: 2 × N × tokens FLOPs over the effective TFLOPS of the replica. The chain that gives a 0.24 s floor on a node, the single-card version, and what it means for time to first token.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

Prefill is the compute-bound half of serving: 2 × N × tokens FLOPs over the effective TFLOPS of the replica. The chain that gives a 0.24 s floor on a node, the single-card version, and what it means for time to first token.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational
🚀 Inference & Serving
Prefill vs DecodeAn LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.
Advanced
🧩 GPU & Accelerator Architecture🔒 Premium
Memory-Bound vs Compute-Bound KernelsEvery kernel is limited by one of two walls: how fast bytes arrive from HBM, or how fast the tensor cores can multiply. Which wall applies is decided by arithmetic intensity against the ridge point, and the two regimes need opposite fixes. Decode, LayerNorm and softmax are memory-bound; prefill GEMMs are compute-bound; the interview question is which one you are looking at and what you would do about it.
Core
🧮 Napkin Math & CapacitySign in
Arithmetic Intensity by OperationThe roofline says a kernel's ceiling is set by its FLOPs per byte against the hardware's ridge point. This page does the FLOPs-per-byte arithmetic for the operations an LLM actually runs (decode at several batch sizes, prefill, the attention score matmul with and without FlashAttention, LayerNorm, an embedding lookup) so the reader can place any of them on the roofline from first principles and say which lever moves it. The numbers explain why a serving fleet's GPUs report 30% utilization while fully loaded.
Advanced
🚀 Inference & Serving🔒 Premium
Disaggregated Prefill and DecodePrefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackEstimate how many H100-hours it took to train Llama 3.1 405B, then check it against the paperNext in this trackSize an inference fleet for a 70B model serving 1,000 concurrent usersNext in this trackA MoE model and a dense model reach the same quality. How do their GPU counts compare for serving and for training?
FEDITOR'S NOTE

The interviewer wants the candidate to switch from bandwidth thinking (decode) to compute thinking (prefill) and to say why, then to produce a TTFT floor with an MFU stated.

DISCUSSION · 0

No comments yet — be the first to share your approach.