Twenty-eight of sixty-four new nodes fail acceptance. What do you do first?

A 44 percent failure rate is far above any plausible hardware defect rate, so the first hypothesis is that the test or the environment is wrong rather than the hardware. The three checks that separate those, the cohort analysis that localizes a real defect, and how to run the vendor conversation.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🖧 Hardware & Cluster Build-Out

Burn-In and Acceptance TestingNew hardware fails early or it fails late, and burn-in exists to move the early failures before the cluster is handed over rather than after. A proper acceptance test runs every layer under sustained load for days, compares every node against its siblings rather than against a specification, and produces a signed number the buyer and the vendor both agree on. The comparison is the important part: identical hardware running identical work should produce identical numbers, and the outliers are the finding.

Foundational

🧭 Ownership & Judgment

Escalation That WorksEscalation has a reputation as a political act because most of it is done badly: a problem handed upward with no options and an implicit request that someone else choose a side. Done well it is a one-page artifact with two or three costed options, a recommendation, the decision needed, a date, and what you will do by default if no answer arrives. That last line is what converts a message into a decision, and it is the part almost everyone omits.

Foundational

🧭 Ownership & Judgment

Deciding Under Incomplete InformationMost infrastructure decisions are made before the evidence is complete, and the skill being assessed is not judgment in the abstract but classification: whether the decision can be undone cheaply. Reversible decisions deserve speed and a scheduled review; one-way decisions deserve the delay and a second opinion. Engineers who apply the same deliberation to both are slow where speed is free and careless where it is not.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Node Health Checks and Burn-InThe cheapest failure is the one caught before a job starts. A GPU fleet runs three kinds of check: a burn-in of days on new or repaired hardware to shake out the parts that die young, a pre-flight suite of minutes before a node is handed to a job, and continuous checks during the job. Each test has an expected number (a GEMM within a few percent of the fleet median, an all-reduce at rated bus bandwidth, a NIC at line rate, a memory test with zero errors), and a node that misses any of them goes to quarantine. This page gives the suites, the numbers, and the arithmetic that justifies the time they cost.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackA GPU cluster has four separate networks. Name them and say what each one carries.Next in this trackDesign the fabric for 10,000 GPUs. Two tiers or three, and what does the choice cost?Next in this trackFor 64 to 72 GPUs, is one GB300 NVL72 rack better than eight HGX B300 nodes?

FEDITOR'S NOTE

Scored on suspecting the test and the environment before the hardware at that rate, on cohort analysis by rack, batch and position, and on escalating with evidence rather than a count.

DISCUSSION · 0

No comments yet — be the first to share your approach.