← 🩺 Fleet Reliability & Observability
Core
DCGM and GPU Telemetry
NVIDIA's Data Center GPU Manager reads a GPU's counters, runs its diagnostics and exports both to the monitoring stack, and nearly every fleet's dashboards and alerts are built on it. The skill is knowing which of its hundreds of fields carry signal: the profiling metrics that say whether the tensor cores are busy (not the utilization number everyone reads first), the error counters that predict a failure, the throttle reasons that explain a slow step, and the diagnostic levels that decide whether a node returns to the pool. This page walks those fields, derives an MFU estimate from them, and gives a fleet's alert thresholds.
a free account unlocks the core curriculum tier · no card
RELATED CONCEPTS
LESSONS THAT TEACH THIS
Foundations of AI InfrastructureObservability for a model service is three signals, not a dashboard14 min→Fleet and ReliabilityDevices fail in a small number of ways, and each one announces itself differently15 min→Fleet and ReliabilityA health gate is what stops one bad node from eating a good job14 min→
PRACTICE THIS IN REAL QUESTIONS
GPU Fleet Reliability & ObservabilityWith DCGM available on every node, what do you actually collect, what do you alert on, and what do you deliberately ignore?→GPU Fleet Reliability & ObservabilityDesign observability for a large training cluster. What do you collect, what does each signal answer, and what pages someone?→LLM Inference & Servingnvidia-smi shows 30 percent utilization on our serving fleet. Is that a problem, and what would you look at instead?→AI Infrastructure System DesignDesign observability for an inference fleet: per-request spans, GPU metrics, and cost attribution per tenant.→GPU Fleet Reliability & ObservabilityDefine the service level objectives for an LLM serving fleet, and the alerting that tells you when one is about to be missed.→GPU Fleet Reliability & ObservabilityHow would you detect that GPUs are thermally throttling, and what is the right response when they are?→
