Meta GPU Fleet Reliability & Observability interview questions

A published run gives the numbers directly: 419 unexpected interruptions in 54 days on 16,384 GPUs, and about three quarters of them hardware. What that implies per GPU-hour, which components dominate, and why a fleet above a few thousand GPUs must treat failure as routine rather than exceptional.

A training run has one number that matters and a handful that explain it. Goodput as the top-level metric, the per-rank timing that finds a straggler among a thousand, the hardware layer beneath it, and the three dashboards that serve three different people asking three different questions.

Correctable errors are corrected, so nothing is wrong yet, and that is exactly why they are the most useful signal a fleet has. What rising rates predict, the remapping budget that decides whether a device can heal itself, and the policy that converts a trend into a scheduled drain instead of a failed training run.

A hang is a collective that one rank never entered, and finding it means asking which rank is missing rather than what is broken. The flight recorder that answers that in seconds, what to do when it is not enabled, and the four causes that a missing rank turns out to be.

Three quantities decide it: how often the job is interrupted, how long a checkpoint takes, and how long a restart takes. The formula that turns them into an optimal interval, the goodput it leaves, and the measured result showing that making checkpoints faster is worth more than checkpointing more often.

Availability is the wrong promise for a platform whose jobs run for weeks. Four objectives that a researcher can actually plan against, how each is measured from data you already collect, the error budget that makes them enforceable, and the one number that should never be an objective.

A link that corrupts occasionally does not fail; it retransmits, and each retransmission costs bandwidth that shows up as a slow job with no error anywhere. The counters that reveal it, the arithmetic turning a replay rate into lost bandwidth, and the threshold where watching becomes draining.

The timeline is the easy part and the detection gap is the valuable part: not what broke, but how long it was broken before anyone knew and why. A worked example with its five sections, the distinction between the trigger and the cause, and the property that separates action items that ship from ones that do not.

The controller's real design problem is not detecting faults but deciding when a signal is strong enough to remove a node from a job. The six states and their transitions, the cost asymmetry that sets every threshold, and the arithmetic showing what a one-percent false-positive rate costs at fleet scale.

The number is easy to state and hard to make honest, because every minute has to be classified and the classification is where the value is. The definition that survives scrutiny, the five categories the missing time falls into, and the property that turns a metric into a work list.

Each rank keeps a ring buffer of its recent collectives, and on a timeout every rank dumps it. Comparing sequence numbers across a thousand of those buffers finds the missing participant in one pass, and the three patterns those comparisons produce point at three different causes.

Error correction catches memory faults and reports them. Nothing catches an arithmetic unit that occasionally returns a wrong product, which is why this class is found by comparing results rather than by reading counters. Three detection strategies with their costs, and the signature in a loss curve.

Every replacement costs a spare, a maintenance window and a return process; every device kept costs the risk of a job it will fail. Four triggers that decide it without a case-by-case argument, the arithmetic that sets the repeat threshold, and why the device's history beats any diagnostic result.

Three separate reserves get merged into one number and then argued about. The repair pipeline from Little's law, the restart pool that has to be instantly available, and the correlated-failure buffer sized by the largest thing that can fail at once, each derived and then added.

Published data gives an interruption rate, and replacements are a fraction of interruptions, so the comparison needs care before it means anything. The derivation that turns one into the other, the answer for a healthy fleet, and the four cuts through the data that turn an anomalous number into a cause.

The driver logs a numbered code when something goes wrong on a GPU, and the number tells you whether to retry the job, drain the node or file a hardware return. The codes worth memorizing in three groups, the action each implies, and the automation that turns a log line into a drained node.

The exporter offers hundreds of fields and about a dozen of them change decisions. The health fields that predict failure, the performance fields that tell you whether work is happening, the one everybody alerts on that means nothing, and the diagnostic levels with the time each takes.

The device has stopped answering on PCIe, so the driver cannot reset it and no software fix exists: the host has to re-enumerate the bus. What causes it, why the node is out until it reboots, the three-strike policy that separates a transient from a dying card, and the automation that handles it without a human.

Five causes produce the same symptom and each has a check that takes under a minute. The order to run them in, what each one eliminates, and the arithmetic showing that one slow node in a thousand costs the whole job its speed rather than a thousandth of it.

Four objectives, each measured at a percentile because averages hide the experience you are promising. The error budget in minutes per month, the burn-rate arithmetic that catches a fast failure in an hour and a slow one in a day, and why two of the four need separate targets per traffic class.

Infant mortality is real and diagnostics that run for a minute do not find it. The five-stage acceptance suite with what each stage catches and how long it takes, the pass criteria as numbers rather than judgments, and the sustained-load stage that finds the faults everything else misses.