With DCGM available on every node, what do you actually collect, what do you alert on, and what do you deliberately ignore?

TL;DR: Collect in two groups with different purposes. Health fields answer "is this device going to fail": ECC single-bit and double-bit counters, pending and failed row remappings, XID events, NVLink CRC and replay counts, memory and die temperatures, power draw and clock throttle reasons. Performance fields answer "is this device doing useful work": tensor pipe activity, memory bandwidth utilization, and SM occupancy. The field to collect and never alert on is GPU utilization, which reports the fraction of time any kernel was resident and reads 100% whether the GPU is at full tensor throughput or running one thread; it is the most commonly alerted and least informative number in the stack. Alert on the health fields with a per-device history rather than an instantaneous threshold, since a rising single-bit rate matters and a single event does not. The diagnostic suite has levels: a quick check in seconds, a medium one in a couple of minutes, and a long one running tens of minutes, which is the one that belongs in node acceptance rather than in a health loop.

How to approach it

Split the fields by the question each answers, because collecting everything and alerting on whatever looks alarming is the default failure. Name the predictive fields specifically. Then the utilization trap, since it is the most common mistake and it is worth being explicit about. Close with the diagnostic levels and where each belongs, because they have very different runtimes and the wrong one in the wrong place either wastes hours or catches nothing.

A strong answer

A typical situation: a cluster has a dashboard of GPU utilization per node and alerts when it drops below 90%. It fires constantly during checkpoints and data-loader stalls, nobody acts on it, and it is silent through a week of rising single-bit errors that ends in an uncorrectable failure during a long run.

The two groups:

HEALTH: does this device need attention?
  ECC volatile and aggregate single-bit counts     rising rate is a leading indicator
  ECC double-bit counts                            any occurrence is a fatal-class event
  pending row remappings                           a row is waiting for a reset to be remapped
  row remap failure                                remapping capacity exhausted: replace
  XID events                                       the fault codes, grouped by required action
  NVLink CRC and replay counters                   rising replays degrade bandwidth first
  GPU die and memory temperatures                  memory temperature is the one that limits
  power draw and clock throttle reasons            distinguishes thermal from power capping
  PCIe replay counters                             a marginal link before it fails

PERFORMANCE: is this device doing useful work?
  tensor pipe activity                             the fraction of cycles the tensor cores ran
  memory bandwidth utilization                     the fraction of peak HBM bandwidth used
  SM occupancy                                     resident warps against the maximum
  these three together tell you whether a job is compute-bound, bandwidth-bound or stalled,
  which utilization cannot

DELIBERATELY NOT AN ALERT: GPU utilization
  what it measures: the fraction of the sampling period during which at least one kernel was
  resident on the device
  what it does not measure: how much of the device that kernel used
  so it reads 100% for a kernel using one SM and for one saturating all 132
  sanity: a decode workload at batch 1 shows 100% utilization while using a few percent of the
          tensor cores, and a data-loader stall shows 0% on a perfectly healthy GPU. Neither
          reading supports a decision, which is why it is collected for capacity conversations
          and never wired to a page

DCGM and GPU Telemetry has the field names and their exact semantics; GPU Failure Modes and XID Errors covers the fault codes the health group carries.

The alert volume decides the thresholds, so compute it before choosing them:

fleet          16,384 GPUs, failure rate about 2e-5 per GPU-hour
events per day = 2e-5 x 16,384 x 24 = 7.9

  fatal class (page)             say 40% of events    = 3.2 pages per day
  recoverable class (ticket)     the remainder        = 4.7 tickets per day

now the false positives, which is where naive thresholds fail:
  a per-device rule that fires at three standard deviations above that device's baseline,
  evaluated once a minute on 16,384 devices
  false positive rate per evaluation, one-sided at 3 sigma = 0.00135
  expected false alerts = 16,384 x 0.00135 = 22 per evaluation interval
  requiring the condition to persist for 10 consecutive intervals:
    0.00135^10 per device, effectively zero, while a real trend persists easily
sanity: three real pages a day is a workable rota and twenty-two false ones an interval is
        not, so the duration requirement is doing more work than the threshold. Any alert
        defined without one will be muted within a week, and then the real events are muted
        with it

What to alert on, and at what threshold:

why history rather than a fleet-wide constant for the ECC rate:
  devices differ. A GPU that has always corrected a few errors per day is not the same as one
  that corrected none for six months and now corrects fifty
  a fleet-wide threshold either fires on the first group constantly or misses the second
  the useful comparison is each device against its own baseline, plus a fleet-wide outlier
  check for devices far above the population

The diagnostic levels, and where each belongs:

level 1   seconds. Basic sanity: the device responds, the driver is loaded, memory is
          addressable. Belongs in a pre-job hook, cheap enough to run before every job.
level 2   about a minute or two. Adds memory and PCIe checks, plus light compute.
          Belongs in the health loop after a suspicious event, before returning a node.
level 3   tens of minutes. Adds sustained stress, a full memory test, and the checks that
          find marginal hardware. Belongs in node acceptance and after a repair.
level 4   longer still, for deep investigation of a specific suspected fault.

the mistake to avoid: running level 3 in a health loop, which takes a node out of service for
half an hour on every transient, or running level 1 at acceptance, which passes hardware that
fails under sustained load an hour later

Node Health Checks and Burn-In covers the acceptance suite this fits into.

The reversal condition: the two-group split assumes a training or serving fleet where the platform owns the hardware and the users own the jobs. On a research cluster where users run varied code, the performance fields become a user-facing product rather than an operations signal: a researcher wants to see their own tensor activity to know whether their code is efficient, and the platform does not want to alert on it at all. Same fields, different consumer, and conflating the two produces dashboards that serve neither.

What interviewers probe next

• "What replaces GPU utilization for capacity questions?" Tensor pipe activity for compute and memory bandwidth utilization for bandwidth, which together say whether a fleet is busy and which resource is the limit.
• "How do you avoid alert fatigue on ECC?" Per-device baselines, ticket rather than page for the correctable class, and a single-bit alert on rate change rather than on absolute count.
• "What is the sampling interval?" Seconds for health counters, since they are cheap and rare, and a similar interval for the profiling fields, which do cost a small amount of device time to collect.
• "Does collecting profiling fields slow jobs?" Slightly, since some require the profiling interface. Measure it once on your workload rather than assuming, and the usual finding is that it is well under a percent.

Common mistakes

• Alerting on GPU utilization, which is 100% for a nearly idle decode step and 0% during a normal data-loader stall.
• Fleet-wide ECC thresholds instead of per-device baselines.
• Running the long diagnostic in the health loop, which removes nodes for half an hour on every transient.
• Collecting the health fields and never building the per-device history that makes them predictive.

Key takeaways

• Two groups: health fields predict failure (ECC, remaps, XID, NVLink, temperature, throttle reasons), performance fields describe work (tensor activity, bandwidth utilization, occupancy).
• GPU utilization is 100% whether a kernel uses one SM or all of them, so it is collected and never alerted on.
• Page on double-bit ECC, remap failure and device-off-bus; ticket on pending remaps and rising correctable rates against each device's own baseline.
• Diagnostic levels run in seconds, minutes and tens of minutes: the quick one before jobs, the long one at acceptance.