Transceivers are failing at three times the rate you budgeted. Find out why.

Modules do not fail randomly at three times a published rate, so the excess has a cause and it is usually thermal or a batch. The cohort cuts that localize it, the counters that separate a dying module from a dirty connector, and the environmental cause that looks like a hardware defect.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🖧 Hardware & Cluster Build-Out

Cables, Transceivers and the Optics Power BudgetCable choice is set by distance and it is the most common ordering mistake in a GPU cluster build. Passive copper reached 3 m at 400G and tops out near 2 m at 800G, so a bill of materials copied from the previous generation produces links that will not come up. Beyond copper come active copper, then active optical cables, then transceivers and fiber. Each step adds reach and adds power, and at cluster scale the transceivers alone draw tens of kilowatts that nobody budgeted.

Foundational

🖧 Hardware & Cluster Build-Out

Direct-to-Chip Liquid Cooling and CDUsAbove roughly 40 kW a rack cannot be cooled by air in any practical hall, which is why every dense GPU deployment now runs liquid to the chip. A cold plate sits on each GPU, a coolant distribution unit isolates the clean rack loop from facility water, and the facility side runs warm, typically 30 to 40 degrees supply, because warm water is cheaper to make. The design numbers are flow rate and temperature rise, and both fall out of one equation that every operator should be able to do from memory.

Foundational

🖧 Hardware & Cluster Build-Out

Scale-Out Fabric Choice: InfiniBand XDR vs Spectrum-XOutside the NVLink domain every GPU talks over a scale-out fabric, and as of September 2026 NVIDIA sells two at the same 800 Gb/s per port: Quantum-X800 InfiniBand and Spectrum-X Ethernet. They differ in congestion handling, operational familiarity and what happens when something misbehaves rather than in headline speed. The switch radix decides how many endpoints a two-tier fabric reaches, and that single number drives the switch count, the cable count and a large part of the budget.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Thermal, Power and Cooling EventsA GPU that gets too hot or is denied power does not fail; it slows down, and on a synchronous job a slow GPU is a slow job. Thermal and power events are the most common cause of the 'nothing failed but the run is 15% slower' ticket, and they are the incidents that scale from one node to a whole hall when a cooling distribution unit or a power feed has a problem. This page explains how throttling works, derives the step-time cost of a clock reduction, walks the failure modes of air and liquid cooling, and covers the power behaviour peculiar to training: thousands of GPUs going idle and busy in lockstep.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the system that tells you every firmware version on every device in the fleet.Next in this trackA vendor offers 15 percent off if you take their Ethernet fabric instead of InfiniBand. Decide.Next in this trackTwo thousand GPUs are being replaced. What do you do with the old ones?

FEDITOR'S NOTE

Scored on cohort analysis over the failure population, on temperature as the leading cause, and on distinguishing module failure from connector and reach problems.

DISCUSSION · 0

No comments yet — be the first to share your approach.