Alert Design and On-Call Load
An alert exists to change what a human does, so any alert that fires without a decision attached is a false alarm regardless of whether its condition was true. GPU fleets generate a specific set of noisy signals that look serious and are not, and separating those from the ones that need a person at three in the morning is what keeps a rotation sustainable. The measure of an alerting system is the fraction of pages that led to an action.
TL;DR: Alert on symptoms a user or a job would notice, not on causes, because causes are numerous and most of them are survived. Page on service level objective burn rate rather than on instantaneous breaches, which distinguishes a blip from a trajectory that will exhaust the error budget. Deduplicate by node and by cause, so a rack losing power produces one page rather than thirty-two. Suppress during declared maintenance, or the maintenance itself generates the noise that trains people to ignore the pager. And know the GPU-specific false alarms, because a fleet generates several signals that look like failures and are normal: a corrected memory error, a driver-level error caused by a user's own kernel, and utilization dropping to zero during a checkpoint. Measure the rotation by actionable pages as a fraction of total, and treat anything below about three quarters as an alerting bug rather than an operations problem.
Symptom, not cause
A fleet has many causes and few symptoms. Pages should sit on the symptoms, with the causes available as context once someone is looking.
| Alert on this symptom | Rather than these causes |
|---|---|
| Training step time above its threshold for N minutes | A degraded NVLink, a thermal cap, a slow rail, a straggler rank |
| Job restart rate per queue above baseline | Any individual node failure |
| Serving p99 latency breaching its objective | Cache pressure, a slow replica, a routing skew |
| Jobs unable to start for N minutes | Quota, capacity, image pull, a health-check loop |
| Fleet capacity below a floor | Individual nodes going out of service |
why cause-based alerting fails on a GPU fleet
at the published failure rate of about 2 x 10^-5 per GPU-hour, a 2,048-GPU fleet sees
2,048 x 24 x 2e-5 = about one hardware event per day
most are absorbed: a job restarts, the scheduler places elsewhere, nobody needed to act
paging on each one produces roughly 30 pages a month for events that required no decision
sanity: the same fleet paging on symptoms produces a page only when a job actually could not
proceed, which is a much smaller number and every one of them has an action
Burn rate rather than instantaneous breach
an objective of 99.5 percent over 30 days
error budget = 0.5 percent of 30 days = 3.6 hours
instantaneous alerting: page whenever the current window is below 99.5 percent
a two-minute blip breaches it and pages, and nothing needed doing
burn-rate alerting: page on how fast the budget is being consumed
a 14.4x burn rate exhausts 30 days of budget in about 50 hours: page immediately
a 6x burn rate exhausts it in about 5 days: page during working hours
a 1x burn rate is the objective being met exactly: no page
the two-window form
require the burn rate to hold over both a long and a short window, so a spike that
recovers does not page and a sustained trend does
sanity: this converts "is it broken right now" into "will this cost us the objective", which
is the question a person being woken can actually act on
SLOs for AI Systems covers defining the objective the burn rate is measured against.
The GPU-specific false alarms
Four signals look like failures on a fleet and usually are not, and an alerting system that has not accounted for them will page constantly.
- Corrected memory errors. Single-bit errors are corrected by design and occur continuously across a large fleet. The alert belongs on the rate rising against its own baseline, or on remapping capacity approaching exhaustion, not on any individual event.
- Driver-level errors caused by user code. Some error classes are raised by a job's own kernel doing something invalid, so they indicate a broken job rather than broken hardware. Routing them to the job owner rather than to the on-call engineer is the fix.
- Utilization dropping to zero. Checkpoints, evaluation phases and data loader stalls all produce it, and a threshold on instantaneous utilization pages during normal operation. Pair it with progress, meaning the step counter, before it means anything.
- A node briefly unreachable. Reboots and agent restarts happen. Require the condition to persist before it becomes a page.
Measuring the rotation
The one metric that governs an alerting system is the fraction of pages that led to an action. Below about three quarters, the problem is the alerting rather than the fleet, and the fix is to delete or downgrade alerts rather than to add people to the rotation. Two supporting measures are worth tracking: pages per shift, since a rotation that regularly exceeds a couple of interruptions per night is not sustainable regardless of the actionable rate, and repeat pages for the same cause, which point at a missing automatic remediation rather than at a missing alert.
What interviewers are listening for
The distinction between symptom and cause, said explicitly, and burn rate rather than instantaneous thresholds. After that, the GPU-specific false alarms, because naming corrected memory errors and user-caused driver errors is what shows a candidate has carried a pager for a fleet rather than for a web service. The last signal is measuring the actionable fraction and treating a low one as an alerting defect, since that reframes on-call load as something the team designed rather than something that happens to it.
Key takeaways
- Alert on symptoms a user or job notices; causes belong in the context, not on the pager.
- Page on error-budget burn rate over two windows, so a blip is silent and a trajectory is loud.
- A 2,048-GPU fleet sees about one hardware event a day that needs no human action, which is why cause-based paging fails.
- Account for the four GPU false alarms: corrected memory errors, user-caused driver errors, utilization dips at checkpoints, and brief unreachability.
- Track the fraction of pages that led to an action; below about three quarters, fix the alerts rather than the rotation.