Cables, Transceivers and the Optics Power Budget
Cable choice is set by distance and it is the most common ordering mistake in a GPU cluster build. Passive copper reached 3 m at 400G and tops out near 2 m at 800G, so a bill of materials copied from the previous generation produces links that will not come up. Beyond copper come active copper, then active optical cables, then transceivers and fiber. Each step adds reach and adds power, and at cluster scale the transceivers alone draw tens of kilowatts that nobody budgeted.
TL;DR: Pick by distance first, then price the power. Passive direct-attach copper is cheapest and draws essentially nothing, and at 800G it reaches about 2 m where the same passive cable reached 3 m at 400G, which is the trap: a bill of materials carried over from a 400G build produces 3 m passive DAC orders that do not link at 800G. Past that come active copper cables, which add a retimer and a few watts, then active optical cables, then pluggable transceivers over fiber. Multimode SR8 optics reach roughly 50 to 100 m and single-mode DR8 reaches 500 m with breakout to smaller lanes. Power matters at scale: vendor documentation puts 400G OSFP modules at roughly 8 to 12 W and twin-port 800G OSFP at about 15 W, and both ends of every link need one. A 512-GPU scalable unit therefore carries thousands of modules and tens of kilowatts of optics, which belongs in the power plan and usually is not.
The ladder, by distance
| Type | Typical reach at 800G | Power per end | Where it is used |
|---|---|---|---|
| Passive DAC | about 2 m | negligible | inside a rack, node to top-of-rack |
| Active copper (ACC or AEC) | a few metres | a few watts | adjacent racks, short intra-row runs |
| Active optical cable (AOC) | tens of metres | similar to a transceiver pair | row to row, fixed length |
| Multimode transceiver (SR8) plus fiber | roughly 50 to 100 m | 8 to 15 W | leaf to spine within a hall |
| Single-mode transceiver (DR8) plus fiber | up to 500 m | 8 to 15 W | across a hall, between halls, with breakout |
The reach numbers come from vendor documentation rather than from a standard, and they move with signal-integrity work, so treat them as the current state of a moving target and check against the specific part number before ordering. The generation-to-generation shrink is the part worth memorizing: higher signalling rates lose reach on copper, every time.
The power nobody budgets
transceivers in one 512-GPU scalable unit
node-to-leaf links (one per GPU): 512
leaf-to-spine links (same count): 512
total links: 1,024
modules needed (one per end): 2,048
power, using vendor figures of about 15 W for a twin-port 800G OSFP module
2,048 x 15 W = 30,720 W = about 31 kW
put it against the compute
16 racks at over 50 kW each = over 800 kW of compute in the same SU
optics as a fraction: 31 / 800 = about 3.9%
sanity: 31 kW is roughly two-thirds of one compute rack's worth of power spent on nothing but
turning electrons into photons, and it lands on the switch rows rather than the
compute rows, which is why it surprises people who budgeted power per GPU
The heat goes somewhere too. A switch with 72 OSFP cages fully populated is dissipating close to a kilowatt of optics on its own faceplate, and that is on top of the switch ASIC. Vendor guidance on airflow around densely populated switch faceplates exists for this reason, and ignoring it produces transceivers running hot, which is one of the mechanisms behind intermittent link errors.
Copper where you can, and why NVL72 is built that way
why a rack-scale system uses a copper backplane for NVLink
the NVLink spine inside a GB300 NVL72 joins 72 GPUs at 130 TB/s aggregate
doing that with optics would need thousands of modules inside one rack
at roughly 15 W per module that is tens of kilowatts added to a rack already at 120 kW+
copper over the short distances inside one rack costs almost no power and no module count
sanity: the copper backplane is not a cost saving so much as a power and reliability
decision, and it is one reason the scale-up domain stops at a rack rather than
spanning a row
The failure modes that trace back to cabling
- A link that will not come up. Almost always the reach: a passive DAC ordered at the previous generation's length. Check the part's rated reach at the rate you are actually running.
- A link that comes up and flaps. Marginal signal integrity, a dirty or damaged connector, or a transceiver running hot. NVLink and Fabric Faults covers the counters that separate these.
- A rail with lower bandwidth than its siblings. One node cabled to the wrong leaf, which a rail-optimized design makes both easy to do and easy to detect, since the rails should be symmetric.
- Errors that appear only under load. Thermal, usually. The transceiver is inside its rating at idle and outside it when the switch is busy and the faceplate is hot.
What interviewers are listening for
Concrete numbers and the awareness that reach shrinks with rate. A candidate who says "we use DAC in the rack and optics between rows, and I would check the 800G reach because passive copper lost about a metre going from 400G" has demonstrated they have actually specified a build. The optics power budget is the second thing, because almost nobody includes it and it is tens of kilowatts at cluster scale. If the conversation is about a hardware fault, the strong move is to separate reach problems, connector problems and thermal problems, since they present similarly and are fixed differently.
Key takeaways
- Choose by distance: passive DAC to about 2 m at 800G, active copper a few metres, AOC or SR8 to roughly 50 to 100 m, DR8 single-mode to 500 m.
- Passive DAC reached 3 m at 400G and about 2 m at 800G, and reusing the old length is the most common ordering mistake.
- Vendor figures put 400G OSFP modules at roughly 8 to 12 W and twin-port 800G at about 15 W, and every link needs two modules.
- A 512-GPU scalable unit needs about 2,048 modules and roughly 31 kW of optics, close to 4% of its compute power.
- Rack-scale NVLink uses a copper backplane because doing it with optics would add thousands of modules and tens of kilowatts inside one rack.