A node fails burn-in twice and passes on the third attempt. Do you accept it?

Two failures and a pass is not a passing node, it is a node with a demonstrated intermittent fault and a small sample. The arithmetic that shows how weak the evidence for acceptance is, what an intermittent fault costs once it is in a training job, and the policy that removes the argument.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🖧 Hardware & Cluster Build-Out

Burn-In and Acceptance TestingNew hardware fails early or it fails late, and burn-in exists to move the early failures before the cluster is handed over rather than after. A proper acceptance test runs every layer under sustained load for days, compares every node against its siblings rather than against a specification, and produces a signed number the buyer and the vendor both agree on. The comparison is the important part: identical hardware running identical work should produce identical numbers, and the outliers are the finding.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Node Health Checks and Burn-InThe cheapest failure is the one caught before a job starts. A GPU fleet runs three kinds of check: a burn-in of days on new or repaired hardware to shake out the parts that die young, a pre-flight suite of minutes before a node is handed to a job, and continuous checks during the job. Each test has an expected number (a GEMM within a few percent of the fleet median, an all-reduce at rated bus bandwidth, a NIC at line rate, a memory test with zero errors), and a node that misses any of them goes to quarantine. This page gives the suites, the numbers, and the arithmetic that justifies the time they cost.

Foundational

🖧 Hardware & Cluster Build-Out

Colocation, Power Contracts and Site SelectionFor most organizations the constraint on deploying GPUs is not the GPUs. It is finding a hall that can deliver 100 kilowatts or more per rack, reject that heat with liquid, and sign a contract for the power years before the hardware exists. Colocation contracts price reserved capacity rather than consumption, cooling capability is what eliminates most sites, and the lead time on new electrical supply is measured in years while GPUs arrive in months.

Foundational

🖧 Hardware & Cluster Build-Out

The Bill of Materials for a Training ClusterA GPU cluster is not a pile of GPUs. A 512-GPU scalable unit built to NVIDIA's DGX SuperPOD B300 reference architecture needs 64 nodes, four separate networks, thousands of transceivers, storage that can absorb a checkpoint burst, a management plane, racks, power distribution and cooling equipment. Writing the list out in order is how a design becomes a purchase order, and the items people forget are the ones that hold up a deployment for weeks.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYour next hardware generation doubles power per rack. Plan the hall's upgrade.Next in this trackYour cluster is going into a colocation facility rather than your own building. What changes?Next in this trackTransceivers are failing at three times the rate you budgeted. Find out why.

FEDITOR'S NOTE

Scored on treating repeated failure as evidence rather than noise, on pricing an intermittent fault in a large job, and on a policy written before the situation arises.

DISCUSSION · 0

No comments yet — be the first to share your approach.