CoreWeave GPU Fleet Reliability & Observability interview questions

A published run gives the numbers directly: 419 unexpected interruptions in 54 days on 16,384 GPUs, and about three quarters of them hardware. What that implies per GPU-hour, which components dominate, and why a fleet above a few thousand GPUs must treat failure as routine rather than exceptional.

The driver logs a numbered code when something goes wrong on a GPU, and the number tells you whether to retry the job, drain the node or file a hardware return. The codes worth memorizing in three groups, the action each implies, and the automation that turns a log line into a drained node.

The exporter offers hundreds of fields and about a dozen of them change decisions. The health fields that predict failure, the performance fields that tell you whether work is happening, the one everybody alerts on that means nothing, and the diagnostic levels with the time each takes.

Correctable errors are corrected, so nothing is wrong yet, and that is exactly why they are the most useful signal a fleet has. What rising rates predict, the remapping budget that decides whether a device can heal itself, and the policy that converts a trend into a scheduled drain instead of a failed training run.

The device has stopped answering on PCIe, so the driver cannot reset it and no software fix exists: the host has to re-enumerate the bus. What causes it, why the node is out until it reboots, the three-strike policy that separates a transient from a dying card, and the automation that handles it without a human.

Infant mortality is real and diagnostics that run for a minute do not find it. The five-stage acceptance suite with what each stage catches and how long it takes, the pass criteria as numbers rather than judgments, and the sustained-load stage that finds the faults everything else misses.

Throttling is the hardware protecting itself, so it produces no error and no failure, only a job that is quietly slower. The two telemetry fields that name it, the arithmetic linking clock to throughput, and why the response differs completely depending on whether the cause is one node or the room.

A fleet of ten thousand GPUs generates enough hardware events to page someone hourly, and a rota that receives them stops reading them within a month. The three tests an alert must pass to page, the arithmetic of a sustainable rota, and the automation that has to exist first.

The controller's real design problem is not detecting faults but deciding when a signal is strong enough to remove a node from a job. The six states and their transitions, the cost asymmetry that sets every threshold, and the arithmetic showing what a one-percent false-positive rate costs at fleet scale.

Four mechanics do most of the work: symptom-based rules rather than cause-based, burn rate rather than thresholds, deduplication so one event is one page, and suppression during known windows. The arithmetic for each, and the rule that keeps the set from growing forever.

A cold plate holds a few tens of grams of water against seven hundred watts, so temperature rises about two degrees a second and the limit arrives in under twenty. What that timescale rules out, the protection layers that act inside it, and the choice that decides whether you lose a job or the hardware.

Every replacement costs a spare, a maintenance window and a return process; every device kept costs the risk of a job it will fail. Four triggers that decide it without a case-by-case argument, the arithmetic that sets the repeat threshold, and why the device's history beats any diagnostic result.

Published data gives an interruption rate, and replacements are a fraction of interruptions, so the comparison needs care before it means anything. The derivation that turns one into the other, the answer for a healthy fleet, and the four cuts through the data that turn an anomalous number into a cause.

A training run has one number that matters and a handful that explain it. Goodput as the top-level metric, the per-rank timing that finds a straggler among a thousand, the hardware layer beneath it, and the three dashboards that serve three different people asking three different questions.

A hang is a collective that one rank never entered, and finding it means asking which rank is missing rather than what is broken. The flight recorder that answers that in seconds, what to do when it is not enabled, and the four causes that a missing rank turns out to be.

Five causes produce the same symptom and each has a check that takes under a minute. The order to run them in, what each one eliminates, and the arithmetic showing that one slow node in a thousand costs the whole job its speed rather than a thousandth of it.

Three quantities decide it: how often the job is interrupted, how long a checkpoint takes, and how long a restart takes. The formula that turns them into an optimal interval, the goodput it leaves, and the measured result showing that making checkpoints faster is worth more than checkpointing more often.

Availability is the wrong promise for a platform whose jobs run for weeks. Four objectives that a researcher can actually plan against, how each is measured from data you already collect, the error budget that makes them enforceable, and the one number that should never be an objective.

Four objectives, each measured at a percentile because averages hide the experience you are promising. The error budget in minutes per month, the burn-rate arithmetic that catches a fast failure in an hour and a slow one in a day, and why two of the four need separate targets per traffic class.