Define TTFT, TPOT and goodput, and tell me how you would measure each one in production.

TL;DR: TTFT is the time from request receipt to the first streamed token and is dominated by queueing plus prefill compute; TPOT is the mean gap between later tokens and is dominated by the decode step time, which rises with batch and context. Goodput is requests per second that meet both SLOs (for example TTFT p95 under 500 ms and TPOT p95 under 50 ms), and it is the number that falls when you push batch past the point where TPOT breaks even though raw tokens per second keeps climbing.

How to approach it

Give the three definitions with the exact clock boundaries, since vague ones lose points fast. Ask what the product is (interactive chat, agent loop, batch summarization), because the SLO pair and the percentile depend on it. Then say you will show how each metric decomposes into engine phases, how you would instrument it, and why goodput is the one to autoscale on.

A strong answer

A typical situation: the platform reports 3,000 tokens a second and the product team says the app feels slow. Both are right, because tokens per second says nothing about what any one user waited for, and the argument cannot resolve until the metrics are defined.

Latency Metrics: TTFT, TPOT and Goodput are the contract between the platform and the product, so the definitions have to be exact:

• TTFT starts when the gateway receives the last byte of the request and stops when the first token leaves the engine (or reaches the client, if you own the network path). It contains queue wait, scheduling delay, prefill compute and the first sampling step.
• TPOT (also inter-token latency) is (total generation time minus TTFT) ÷ (output tokens minus 1). It contains decode step time, which is bandwidth-bound and grows with the running batch and the KV read.
• Goodput is the count of requests per second whose TTFT and TPOT are both inside the SLO, divided by wall time. A request that finishes with TTFT 400 ms and TPOT 80 ms against a 50 ms TPOT SLO counts as zero.

The decomposition is what lets you predict them:

inputs: Llama 3.1 70B bf16 on 8 × H100 SXM, prefill MFU 0.4
TTFT ≈ queue wait + prompt tokens × 2 × params ÷ (GPUs × peak × MFU)
  1,000-token prompt, empty queue:
    = 0 + 1,000 × 1.41e11 ÷ (8 × 989e12 × 0.4) = 1.41e14 ÷ 3.16e15 ≈ 45 ms
  8,000-token prompt: ≈ 357 ms
  add one queued 8k prompt ahead of it: ≈ 357 + 357 ≈ 714 ms

TPOT at batch B, 4k context ≈ (141.2 GB + B × 1.34 GB) ÷ 26.8 TB/s
  B = 16:  (141.2 + 21.4) ÷ 26.8 ≈ 6.1 ms
  B = 128: (141.2 + 172) ÷ 26.8 ≈ 11.7 ms
  B = 256: (141.2 + 343) ÷ 26.8 ≈ 18.1 ms
sanity: TTFT is set by prompt length and queue, TPOT by batch and context; they fail for different reasons

Goodput turns those into one operating decision. Raw throughput at batch 256 is about 14,000 tokens/s, versus 11,000 at batch 128. If the TPOT SLO is 15 ms at p95, the batch-256 point fails every request and goodput is zero while the tokens/s dashboard looks best. The operating point is the largest batch at which the p95 of TPOT stays under the SLO, with the TTFT budget consumed by queue wait. That trade is the throughput-latency curve every serving team draws.

Measurement details that separate a senior answer:

• Record per-request timestamps at the gateway: t_received, t_first_token, t_last_token, prompt_tokens, output_tokens. Compute TTFT and TPOT per request, never from aggregate counters.
• Report percentiles, and pick them by product: p50 says nothing about the users who complain. Chat products usually run TTFT p95 and TPOT p95 or p99; batch pipelines care about throughput and a loose p99.
• Take TPOT as a mean per request but watch the inter-token histogram separately, because a request with 20 ms average and one 800 ms stall (a prefill of someone else's 20k prompt) has a fine TPOT and a visible freeze.
• Engines export the pieces: vLLM's Prometheus endpoint has time-to-first-token and time-per-output-token histograms, plus running, waiting, KV utilization and preemption counters; join those with gateway timestamps to split queue wait from prefill.
• Define goodput with a window (per minute) and use it as the autoscaler's signal, since it drops before either raw latency percentile alone crosses the line.

The decision: SLO on TTFT p95 and TPOT p95, size batch to the TPOT line, autoscale on goodput. The reversal condition: offline batch jobs, where neither latency matters and Cost per Million Tokens is the only metric.

What interviewers probe next

• "Why not one end-to-end latency SLO?" A 2,000-token answer at 20 ms per token takes 40 s and is fine if it streams; end-to-end latency would flag it while TTFT and TPOT say the user experience is right.
• "What makes p99 TTFT spike while p50 stays flat?" Queue wait behind long prompts, or a prefix-cache miss on a replica that was just restarted; p50 requests never see the queue.
• "How do you report TPOT for a request that emitted one token?" You cannot; exclude it, and track the share of such requests separately so short answers do not silently improve the number.

Common mistakes

• Starting the TTFT clock at engine admission, which hides the queue.
• Measuring TPOT as output tokens ÷ total time, which folds prefill into decode.
• Reporting tokens per second as the headline while goodput is falling.
• Averaging TPOT across requests with different context lengths and calling it a capacity number.

Key takeaways

• TTFT = queue + prefill; prefill for 70B on 8 H100s is about 45 ms per 1,000 prompt tokens at MFU 0.4.
• TPOT = decode step time; 6 ms at batch 16, 12 ms at batch 128, 18 ms at batch 256 for 70B at 4k context.
• Goodput counts requests meeting both SLOs; it is the autoscaling signal and it can fall while tokens/s rises.
• Instrument per request at the gateway; percentiles by product; watch the inter-token histogram for stalls.