Scale-Out Fabric Choice: InfiniBand XDR vs Spectrum-X

TL;DR: Both options run at 800 Gb/s per port as of September 2026, so speed is not the decision. InfiniBand arrives as a managed fabric with credit-based flow control, a subnet manager and in-network reduction, and it is what NVIDIA's own reference architectures are built around. Spectrum-X is Ethernet with the congestion-control and telemetry work that makes Ethernet behave on AI traffic, and it wins when the organization already runs Ethernet everywhere and wants one operational model. The number that actually shapes the build is switch radix: the Quantum-X800 Q3400-RA offers 144 ports of 800 Gb/s in 4U, and a two-tier fat tree built from radix-R switches reaches R squared over 2 endpoints, which is 10,368 here. Below that, two tiers; above it, three tiers with the extra hop, the extra optics and the extra cost. NVIDIA's DGX SuperPOD B300 reference architecture uses a scalable unit of 64 nodes and 512 GPUs with 8 leaf switches, and that ratio is the practical starting point.

The two options, compared on what differs

The honest summary is that InfiniBand is the lower-risk technical choice for a dedicated training fabric and Spectrum-X is the lower-risk organizational choice for a team that runs Ethernet everywhere else. Congestion Control for AI Fabrics covers why Ethernet needs the extra work at all, and RDMA, InfiniBand and RoCEv2 covers the transport underneath both.

Radix decides the shape of the whole build

a two-tier fat tree from switches with R ports
  each leaf spends half its ports downward to endpoints and half upward to spines
  endpoints per leaf = R/2
  number of leaves that can be fully connected = R (each leaf uses R/2 uplinks, one per spine
    group), so
  max endpoints = R x R / 2

for the Quantum-X800 Q3400-RA at R = 144 (144 ports of 800 Gb/s in 4U)
  max endpoints = 144 x 144 / 2 = 10,368
  at 8 NICs per node that is 1,296 nodes, or 10,368 GPUs on a rail-optimized design where
    each GPU owns one port
sanity: a 8,192-GPU cluster fits in two tiers on 144-port switches and would need three tiers
        on 64-port switches, which is one extra hop of latency and roughly a third more
        optics, so radix is a budget decision and not a detail

Rail-optimized, and why every design uses it

A node has eight GPUs and eight NICs. Rail-optimized means NIC i on every node connects to leaf switch i, so all the rank-i GPUs in the cluster share one switch layer. The payoff is that a collective which communicates between the same rank on different nodes, which is what data-parallel all-reduce does, stays within one rail and never touches the spine.

what rail-optimized buys, in numbers
  a data-parallel all-reduce among rank-0 GPUs on 64 nodes:
    all traffic stays on rail 0's leaf switches, one hop, no spine
  the same collective on a non-rail-optimized design:
    traffic spreads across leaves and crosses the spine, adding a hop and competing with
    every other rail's traffic
  a hop of switch latency is small, on the order of hundreds of nanoseconds, and the real
    gain is contention: keeping eight independent rails from sharing one oversubscribed path
sanity: rail-optimized does not make any single link faster, it stops eight collectives from
        colliding, which is why its benefit shows up as tail latency rather than as bandwidth

Sizing from NVIDIA's reference architecture

NVIDIA's DGX SuperPOD B300 reference architecture, published in their documentation, uses a scalable unit of 64 DGX B300 nodes and 512 GPUs served by 8 leaf switches, with four nodes to a rack and rack power above 50 kW. Four scalable units make 256 nodes and 2,048 GPUs. Using that as the unit rather than inventing one has a practical benefit beyond correctness: the cable counts, rack layouts and switch placements are already worked out, and deviating from them means redoing that work.

one scalable unit, from the reference architecture
  nodes                64 DGX B300
  GPUs                 512
  leaf switches        8, which is one per rail
  node-to-leaf links   512, one per GPU
  racks                16, at 4 nodes per rack
  compute power        16 racks x >50 kW = over 800 kW before storage, management or cooling
sanity: 512 GPUs and 8 leaves means 64 ports used per leaf downward, which fits comfortably
        inside a 144-port switch and leaves the rest for uplinks, so one SU does not even
        fill a leaf

What interviewers are listening for

Whether you can say what differs rather than which is better. Both run at 800 Gb/s; a candidate who answers "InfiniBand is faster" is repeating something that was more true a few generations ago. The strong answer names the operational difference, gives the radix arithmetic that decides two tiers against three, and mentions rail-optimized with the reason it exists. Interviewers at clouds also probe multi-tenancy, because partitioning an InfiniBand fabric and segmenting an Ethernet one are different problems with different tools.

Key takeaways

• As of September 2026 both fabrics run 800 Gb/s per port, so the decision is congestion behavior, operational familiarity and multi-tenancy rather than speed.
• Two-tier fat tree capacity is R squared over 2 endpoints: 10,368 on a 144-port Quantum-X800 switch, against 2,048 on a 64-port switch.
• Rail-optimized connects NIC i of every node to leaf i, keeping same-rank collectives on one rail and off the spine.
• NVIDIA's DGX SuperPOD B300 scalable unit is 64 nodes, 512 GPUs, 8 leaf switches, 4 nodes per rack, over 50 kW per rack.
• Rail-optimization improves tail latency by removing contention rather than raising any single link's bandwidth.