TL;DR: A training platform's users do not care whether the control plane was up; they care whether their run made progress. Four objectives capture that. Goodput: the fraction of wall-clock time a running job spends making forward progress, targeted at 90% and measured from the job's own step accounting. Time to restart: how long from an interruption to the job taking steps again, targeted under 10 minutes and measured from the lifecycle events. Queue wait by priority: how long a submitted job waits for resources, with a different target per tier, measured at submission and start. And checkpoint durability: the probability that the most recent complete checkpoint is readable, which is a promise rather than a percentile and is verified by periodically restoring one. Each gets an error budget over a rolling window, which is what converts an objective into something that changes behavior: when the budget is spent, risky changes stop until it recovers. The number that should never be an objective is cluster utilization, because maximizing it directly conflicts with every objective above.
How to approach it
Start by rejecting availability, since it is the reflex answer and it does not describe what a long-running job needs. Propose objectives that a researcher would recognize as their own experience. Define each with its measurement, because an objective without a measurement is a slogan. Add the error budget, which is the enforcement mechanism. Close with the anti-objective, since naming what not to promise is as useful as the list.
A strong answer
A typical situation: a platform team reports 99.9% availability for the quarter and a researcher points out that their three-week run restarted eleven times, lost about four days of progress, and finished late. Both statements are true. The platform measured whether its services answered requests, and the researcher experienced whether their job made progress, and those are different questions.
The four objectives:
1. goodput
definition useful step time / wall-clock time, per running job
target 90% for jobs above a size threshold
measurement from the job's own step timestamps plus its lifecycle events; the loss
breakdown (restarts, checkpoint pauses, stragglers, loader stalls) must sum
to the gap, which is what makes it auditable
why this one it is the single number a researcher would choose if allowed only one
2. time to restart
definition from an interruption to the job executing steps again
target under 10 minutes at p90
measurement lifecycle events: interruption detected, resources allocated, process start,
checkpoint loaded, first step completed
why it matters at a 3-hour mean time between interruptions, every minute of restart is
0.55% of goodput, so this objective and the first are arithmetically linked
3. queue wait by priority
definition from submission to first step, by tier
target different per tier: interactive under 5 minutes, standard under 2 hours,
batch best effort
measurement scheduler events, reported as a distribution rather than a mean, since the
tail is what researchers remember
why it matters it is the part of the experience the platform controls most directly and
the part most often left unmeasured
4. checkpoint durability
definition the most recent checkpoint marked complete is readable and correct
target stated as a promise, not a percentile: no run loses more than one checkpoint
interval to a storage failure
measurement a periodic restore test: pick a recent checkpoint, load it on a small node
set, verify the tensors and the metadata
why it matters it is the only objective whose failure is unbounded, since a lost checkpoint
can cost days rather than minutes
SLOs for AI Systems covers the general framework; MFU and HFU is the efficiency metric that sits beside goodput and is not an objective, because it depends on the researcher's model and code as much as on the platform.
The error budget, which is what makes an objective enforceable:
example, goodput at 90% over a rolling 30-day window
budget = 10% of wall-clock time may be lost
in a 30-day month = 72 hours of allowed loss per job
consumed by restarts, checkpoint pauses, stragglers, maintenance windows
what the budget does
budget remaining -> the platform may take risks: roll firmware, drain nodes for
maintenance, deploy scheduler changes
budget exhausted -> those stop until it recovers. No new rollouts, maintenance deferred,
and the team works on whatever consumed it
why this is the mechanism rather than the number
an objective with no consequence is a dashboard. The budget creates a decision rule that
both the platform team and the researchers understand in advance, so a maintenance window
during a healthy month is uncontroversial and the same window during a bad one is not
sanity: the budget is deliberately generous at 72 hours per month, because a platform that
never spends it is over-cautious and shipping too slowly
The anti-objective:
cluster utilization should not be an SLO
why: maximizing it conflicts directly with every objective above
spare capacity for fast restarts lowers utilization
a warm pool for interactive queue wait lowers utilization
draining a degraded node lowers utilization
headroom for the next large job lowers utilization
what to do instead: report utilization as a capacity-planning input, track it, and explain
the gap between it and 100% as the sum of deliberate reserves rather than as waste
sanity: a platform pushed to 95% utilization will meet that target by removing exactly the
reserves that its other objectives depend on, and the failure will appear as missed goodput
a quarter later
Capacity Planning and Utilization is where utilization belongs.
The reversal condition: these objectives assume large, long-running jobs, which is the training case. A platform serving many short jobs, such as evaluation sweeps or hyperparameter searches, should promise job success rate and queue wait instead, because goodput per job is noise across thousands of short processes and restart time matters less when a job is ten minutes long. Same instrumentation, different aggregation, and a platform serving both needs both sets rather than a compromise that fits neither.
What interviewers probe next
- "Why not availability?" Because a job runs for weeks and cares about progress, not about whether an API answered. A control plane can be at 99.99% while every job restarts hourly.
- "How do you attribute a goodput loss?" The breakdown must sum to the gap: restarts, checkpoint pauses, stragglers, loader stalls, maintenance. If it does not sum, the instrumentation has a hole rather than the cluster having a mystery.
- "Who owns the budget?" Both sides: the platform team spends it on changes, and the researchers see it, which is what makes a maintenance window a negotiation rather than a surprise.
- "What if a researcher's own code causes the loss?" Attribute it in the breakdown. Loss from a job's own inefficiency is real and is not the platform's budget, which is why the categories matter as much as the total.
Common mistakes
- Promising availability, which measures the platform's services rather than the researcher's experience.
- Objectives with no error budget, which are dashboards rather than commitments.
- Making utilization a target, which systematically removes the reserves the other objectives need.
- A goodput number with no breakdown, which cannot be acted on when it drops.
Key takeaways
- Four objectives: goodput at 90%, time to restart under 10 minutes at p90, queue wait by tier, and checkpoint durability verified by restore.
- Every minute of restart time is 0.55% of goodput at a 3-hour mean time between interruptions, which links the first two arithmetically.
- The error budget, about 72 hours per job per month at a 90% target, is what converts an objective into a decision rule about rollouts and maintenance.
- Never make utilization an objective; report it, and explain the gap to 100% as deliberate reserves.