Design the alerting rules for a GPU platform so that a page is always worth waking for. What are the mechanics?

Four mechanics do most of the work: symptom-based rules rather than cause-based, burn rate rather than thresholds, deduplication so one event is one page, and suppression during known windows. The arithmetic for each, and the rule that keeps the set from growing forever.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🩺 Fleet Reliability & Observability

Alert Design and On-Call LoadAn alert exists to change what a human does, so any alert that fires without a decision attached is a false alarm regardless of whether its condition was true. GPU fleets generate a specific set of noisy signals that look serious and are not, and separating those from the ones that need a person at three in the morning is what keeps a rotation sustainable. The measure of an alerting system is the fraction of pages that led to an action.

Core

🩺 Fleet Reliability & ObservabilitySign in

SLOs for AI SystemsA service level objective is a promise with a number attached, and AI systems need their own because the classic ones do not fit: a training run has no requests, only progress, so its objective is goodput; an LLM endpoint streams, so its latency is two numbers (time to first token and time per token) rather than one; and both spend a budget that is set by hardware failure rates rather than by software bugs. This page defines the objectives that fleet and serving teams actually use, derives the thresholds from user needs and from the hardware, and works the error-budget arithmetic that decides when to stop shipping and start fixing.

Advanced

📐 AI Systems Design🔒 Premium

Evaluation and Data Pipeline InfrastructureBehind every model release is a pipeline that turns raw text into training shards and a harness that runs thousands of evaluation prompts against every checkpoint, and both are infrastructure problems with GPU-sized budgets. The data side is a batch system: dedup, filter, tokenize and shard petabytes with lineage. The eval side is a serving system in disguise: run a benchmark suite against a checkpoint in minutes, on shared GPUs, reproducibly, with results a researcher can trust. This page designs both, derives the compute and storage they need, and gives the reproducibility rules that separate a real harness from a script.

Core

🩺 Fleet Reliability & ObservabilitySign in

DCGM and GPU TelemetryNVIDIA's Data Center GPU Manager reads a GPU's counters, runs its diagnostics and exports both to the monitoring stack, and nearly every fleet's dashboards and alerts are built on it. The skill is knowing which of its hundreds of fields carry signal: the profiling metrics that say whether the tensor cores are busy (not the utilization number everyone reads first), the error counters that predict a failure, the throttle reasons that explain a slow step, and the diagnostic levels that decide whether a node returns to the pool. This page walks those fields, derives an MFU estimate from them, and gives a fleet's alert thresholds.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackCoolant flow to a rack stops. What happens in the next sixty seconds, and what has to be automatic because a human cannot act in time?Next in this trackTrace one request through an inference platform. What are the spans, what does each cost, and where does latency actually hide?Next in this trackOne serving replica has a per-token latency 40 percent worse than its peers. Find out why.

FEDITOR'S NOTE

Scored on symptom-based over cause-based alerting, on burn-rate windows with their arithmetic, on deduplication and suppression as mechanics rather than culture, and on a lifecycle that removes alerts.

DISCUSSION · 0

No comments yet — be the first to share your approach.