How do you measure effective training time, and where does the missing ten percent of a well-run cluster actually go?

The number is easy to state and hard to make honest, because every minute has to be classified and the classification is where the value is. The definition that survives scrutiny, the five categories the missing time falls into, and the property that turns a metric into a work list.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🧮 Open Weights & Serving Engines

Serving Benchmarks That Do Not LieMost published serving numbers are not comparable to each other and not predictive of production, because they differ in the input distribution, the concurrency, whether the cache was warm, and which of several very different metrics is being reported. A benchmark that supports a decision has to fix all four, report a distribution rather than a mean, and be run against the traffic shape you actually serve. The single most useful discipline is to compute the bandwidth bound first, so you know what fraction of the possible you achieved.

Core

🩺 Fleet Reliability & ObservabilitySign in

SLOs for AI SystemsA service level objective is a promise with a number attached, and AI systems need their own because the classic ones do not fit: a training run has no requests, only progress, so its objective is goodput; an LLM endpoint streams, so its latency is two numbers (time to first token and time per token) rather than one; and both spend a budget that is set by hardware failure rates rather than by software bugs. This page defines the objectives that fleet and serving teams actually use, derives the thresholds from user needs and from the hardware, and works the error-budget arithmetic that decides when to stop shipping and start fixing.

Foundational

🩺 Fleet Reliability & Observability

Training Uptime and Interruption StatisticsA frontier training run is a months-long job on tens of thousands of parts, each of which fails rarely, so the run as a whole fails constantly. The numbers that describe this are simple and worth deriving from scratch: a per-GPU failure rate, multiplied by the fleet, gives the interval between stops; the checkpoint interval and the restart time give the work lost per stop; together they give effective training time, which is the fraction of wall-clock the run is actually learning. This page derives all of it from the one public dataset that reports it and shows how to size checkpointing so the run stays above 90%.

Foundational

🚀 Inference & Serving

Latency Metrics: TTFT, TPOT and GoodputAn LLM request has two latencies, not one: time to first token, set by queueing and prefill, and time per output token, set by the decode loop. Reporting them as percentiles, and reporting goodput (requests that met both SLOs per second) rather than raw throughput, is what separates a serving engineer from a benchmark reader. The numbers a loop expects: about 24 tokens per second single-stream for a 70B model on one H100, TTFT floors in the hundreds of milliseconds for long prompts, and p99s that come from queueing, not from the GPU.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWalk me through reading a collective flight-recorder dump. What is in it, and how do you find the rank that caused a hang?Next in this trackThe facility asks you to cap GPU power by 30 percent for the summer. What does that cost a training run, and how would you do it?Next in this trackA GPU computes the wrong answer and reports no error. How would you detect that, and what does it look like in a training run?

FEDITOR'S NOTE

Scored on a definition with a clear numerator and denominator, on the breakdown summing to the gap as the correctness property, and on the instrumentation each category requires.

DISCUSSION · 0

No comments yet — be the first to share your approach.