You own the on-call rota for a GPU fleet. What is allowed to wake someone at 3am, and what must not?

A fleet of ten thousand GPUs generates enough hardware events to page someone hourly, and a rota that receives them stops reading them within a month. The three tests an alert must pass to page, the arithmetic of a sustainable rota, and the automation that has to exist first.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🩺 Fleet Reliability & Observability

Alert Design and On-Call LoadAn alert exists to change what a human does, so any alert that fires without a decision attached is a false alarm regardless of whether its condition was true. GPU fleets generate a specific set of noisy signals that look serious and are not, and separating those from the ones that need a person at three in the morning is what keeps a rotation sustainable. The measure of an alerting system is the fraction of pages that led to an action.

Foundational

🧭 Ownership & Judgment

Escalation That WorksEscalation has a reputation as a political act because most of it is done badly: a problem handed upward with no options and an implicit request that someone else choose a side. Done well it is a one-page artifact with two or three costed options, a recommendation, the decision needed, a date, and what you will do by default if no answer arrives. That last line is what converts a message into a decision, and it is the part almost everyone omits.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Incident Response for GPU FleetsAn incident on a GPU fleet is a training run that stopped, a serving endpoint burning its error budget, or a fleet-wide symptom nobody has explained yet. The response has a shape: detect, stabilize, diagnose, repair, return through the gate, write it up. The stabilizing move (drain the node, restart from checkpoint, or shift traffic) comes before the diagnosis, because a frontier run loses more per minute than any investigation is worth. This page gives the triage order, the 3am decision tree, the spare-capacity arithmetic behind drain-and-replace, and what a fleet postmortem has to contain.

Foundational

🧭 Ownership & Judgment

Deciding Under Incomplete InformationMost infrastructure decisions are made before the evidence is complete, and the skill being assessed is not judgment in the abstract but classification: whether the decision can be undone cheaply. Reversible decisions deserve speed and a scheduled review; one-way decisions deserve the delay and a second opinion. Engineers who apply the same deliberation to both are slow where speed is free and careless where it is not.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the controller that decides whether a GPU node is healthy. What are its states, and how do you tune its false-positive rate?Next in this trackError rate on an inference fleet tripled ten minutes after a deploy. What do you do first, and what should have caught it?Next in this trackHow do you measure effective training time, and where does the missing ten percent of a well-run cluster actually go?

FEDITOR'S NOTE

Scored on the three tests (customer impact, human action required, urgency), on the page-volume arithmetic at fleet scale, and on automation as the precondition rather than an improvement.

DISCUSSION · 0

No comments yet — be the first to share your approach.