TL;DR: Write the criteria before the hardware arrives and make every one of them a number with a tolerance. Four levels of test, run in order so a failure stays attributable: per-GPU diagnostics and memory stress, per-node collectives at sustained power, per-rail pairwise bandwidth across all eight rails, then a full-scale sustained job for 48 to 72 hours. Judge against the fleet median rather than the datasheet, because "within specification" is a weak claim and identical hardware running identical work should agree closely, so a 3 percent tolerance around the median is both testable and meaningful. Duration matters as much as the tests: thermal problems need hours at steady state and a twenty-minute run finds none of them. Include a defect budget agreed in advance, a memory-health clause so a part that has consumed much of its row-remapping capacity is replaced rather than accepted, and a record of the environment, because a test passed with the hall cold is not a test.
How to approach it
Say the criteria are written and agreed before delivery, because that is what makes them enforceable. Then the four levels and what each catches. Then the comparison rule, which is the technical core. Then duration and environment. Then the commercial clauses: defect budget, replacement turnaround, and what happens if the fleet passes on averages and fails on outliers. Close with what you would sign and what you would not.
A strong answer
A typical situation: 64 nodes are racked and powered, the vendor's engineer has run their diagnostic suite and reports all pass, and payment terms are tied to acceptance. The diagnostic suite tests each GPU against its specification and every GPU is inside it. Four weeks later a training job is 6 percent slower than expected and the cause is four nodes in one rack.
The four levels, and the criterion for each:
| Level | Test | Acceptance criterion |
|---|---|---|
| Per GPU | Vendor diagnostics at the deepest level, plus a memory stress pass | All pass; no uncorrected errors; row-remap consumption below an agreed threshold |
| Per node | nccl-tests all-reduce across the node's 8 GPUs; a sustained compute loop at full power for 1 hour | Bus bandwidth within 3 percent of the fleet median; sustained clock within 3 percent of the median; no throttling events |
| Per rail | Pairwise bandwidth on each of the 8 rails, node to node | Every rail within 3 percent of the rail median; no link below its rated width or speed in ibdiagnet |
| Full scale | A representative training job across all 512 GPUs, 48 to 72 hours | Step time stable within 3 percent over the run; no unplanned interruptions beyond the agreed defect budget; thermal and power at steady state recorded |
The comparison rule is the technical core and is worth stating explicitly:
why the fleet median beats the datasheet
a datasheet gives a peak nothing reaches, so "within specification" admits a wide band
512 identical parts running identical work should agree closely
the useful test is therefore the spread, not the level
what the outliers mean
one node 6% low one weak NVLink, or one GPU clocking lower than its peers
four nodes 6% low, same rack a rack-level cause: a cooling loop, a power feed, an
inlet temperature
one rail 15% low cabling, a transceiver, or a node cabled to the wrong leaf
a long tail with no cluster a batch quality problem, which is a commercial conversation
apply the same rule to every measurable
bus bandwidth per node, sustained clock per GPU, power draw at a fixed workload per GPU,
HBM temperature at steady state per GPU, step time per rank in the full-scale job
sanity: on 512 supposedly identical GPUs the interesting finding is almost never a hard
failure. It is the four running 6% slow, and that finding only exists if the
criterion is a spread rather than a floor
Burn-In and Acceptance Testing covers the reasoning behind the levels, including the bathtub curve that makes early testing worth doing at all. ECC, Row Remapping and Memory Errors covers the memory clause, which deserves its own line in the document.
Why duration is a criterion rather than a detail:
what a short test misses
thermal steady state in a liquid-cooled rack takes tens of minutes to reach and longer for
the facility loop to settle
a marginal cold plate, a partially blocked channel or an inlet temperature at the edge of
spec only shows once the whole loop is hot
infant mortality is a rate, so finding it requires GPU-hours
what 72 hours on 512 GPUs actually buys
GPU-hours = 512 x 72 = 36,864
at the corpus steady-state rate of about 2 x 10^-5 per GPU-hour, expected failures =
36,864 x 2e-5 = 0.7
observed counts during burn-in are routinely several times that, and the excess is the
infant mortality being drained onto the vendor's account rather than onto a training run
sanity: if a burn-in finds only the steady-state number, either the batch is unusually good
or the test is not stressing the hardware, and the second is more likely
The commercial clauses, which are what make it an acceptance document:
- A defect budget. Agree in advance how many early failures are covered and the replacement turnaround, rather than negotiating while the cluster is half-broken.
- An outlier clause. A fleet that passes on averages and has a tail is not accepted, and the tolerance is named.
- A memory-health clause. A GPU that has consumed a large share of its row-remapping capacity during burn-in is replaced even though it currently works.
- The environment recorded. Inlet water temperature, hall temperature and the power feed configuration, because these change the result.
- Re-test on replacement. A swapped node re-runs levels one through three before it counts as accepted.
The reversal condition: for a small delivery, say four nodes rather than sixty-four, the fleet-median rule loses its power because four samples do not establish a median worth comparing against. There the comparison has to be against a reference: a known-good node from the existing fleet running the identical test, or the vendor's published number for that configuration with a stated tolerance. It is worth saying this out loud in an interview, because reciting the median rule for a delivery of four nodes shows the method was memorized rather than understood.
What interviewers probe next
- "Why 3 percent?" It is wide enough to absorb normal variation between identical parts and narrow enough to catch a single degraded link. State it as a chosen number and be ready to justify the choice with the observed spread.
- "What if the vendor disputes the median rule?" Offer both: their specification as a floor and the median rule as the outlier test. A vendor confident in the batch has no reason to object to a spread criterion.
- "What do you do with the failures?" Replace, then re-run levels one to three on the replacement, and track whether replacements cluster by rack or by batch.
- "How long before you sign?" After the full-scale run completes and the outlier analysis is done, not when the last test starts passing.
Common mistakes
- Accepting on "all diagnostics passed", which tests each part against a wide specification and finds nothing.
- Writing criteria after the hardware arrives, which makes them a negotiation rather than a contract.
- A test short enough that thermal steady state is never reached.
- No outlier clause, so a fleet with a tail passes on averages.
- Applying the fleet-median rule to a delivery too small for a median to mean anything.
Key takeaways
- Four levels in order: per GPU, per node, per rail, then 48 to 72 hours at full scale.
- Compare against the fleet median with a stated tolerance such as 3 percent, not against the datasheet.
- 72 hours on 512 GPUs is 36,864 GPU-hours, which predicts 0.7 steady-state failures, so finding several times that is the point.
- Include a defect budget, an outlier clause, a memory-health clause, the recorded environment, and re-test on replacement.
- For a small delivery, compare against a known-good reference node, because a handful of samples has no useful median.