InfiniBand or RoCE version 2 for a new GPU training cluster. Make the call and say what would reverse it.

TL;DR: Both deliver RDMA at 400 Gb/s per port and both can run a large training job well. The difference is where losslessness comes from. InfiniBand is credit-based: a sender only transmits when the receiver has advertised buffer space, so packets are not dropped under congestion and the fabric behaves the same on day one and after a year of changes. RoCE version 2 runs RDMA over Ethernet and gets the same property only when priority flow control and explicit congestion notification are configured correctly on every switch and every NIC, and a single mismatched setting produces either packet loss that collapses RDMA throughput or a pause cascade that stalls the fabric. In exchange, RoCE uses the Ethernet team, the Ethernet supply chain and Ethernet tooling, which for many organizations is the deciding factor. For a first training cluster under a few thousand GPUs with no fabric specialist on staff, choose InfiniBand. Reverse it when you have network engineers who own the switches, when you are large enough that the per-port saving funds them, or when the cluster must share a fabric with the rest of the company.

How to approach it

Name the one technical difference that matters, which is where losslessness comes from, and resist listing features. Then convert the difference into operational terms, because that is where it is actually paid. Then give the cost and staffing side honestly. Make the call for a stated situation, and give the two conditions that flip it, since the answer depends on the organization rather than on the technology.

A strong answer

A typical situation: a team stands up a RoCE cluster, runs nccl-tests on two nodes and sees line rate, then starts a 512-GPU job and watches all-reduce bandwidth collapse to a fraction of that with no errors reported anywhere. The two-node test never produced congestion, so it never exercised the configuration that was wrong.

The one difference that generates all the others:

InfiniBand
  credit-based flow control at the link layer: a sender transmits only into buffer space the
  receiver has already advertised. Congestion becomes backpressure, never loss.
  A subnet manager assigns addresses and computes routes centrally, so the fabric has one
  authoritative view of itself.

RoCE v2
  RDMA packets inside UDP over Ethernet. Ethernet drops under congestion by default, and RDMA
  reacts badly to loss because go-back-N retransmission at these rates destroys throughput.
  Losslessness is added by configuration:
    priority flow control (PFC) so a congested port pauses its upstream instead of dropping
    explicit congestion notification (ECN) plus DCQCN so senders slow down before PFC fires
    buffer and threshold tuning per switch model
  every switch, every port, both NICs, consistently. One wrong threshold and you get either
  loss or a pause cascade

Congestion Control for AI Fabrics covers the mechanisms; RDMA, InfiniBand and RoCEv2 covers the transports. The operational consequence is the part to say out loud: on InfiniBand the correct configuration is mostly the default, and on RoCE the correct configuration is a project with an owner and a regression test.

What the choice costs and saves:

The arithmetic that decides it at a given size:

cluster    1,024 GPUs, one 400 Gb/s port per GPU, plus storage and management
ports      1,024 endpoint ports plus roughly 512 uplinks at 1:1 = about 1,536 fabric ports
cost delta assume $2,000 per port InfiniBand against $1,500 RoCE, including optics
           1,536 x $500 = about $768,000 of capital saved by choosing RoCE
against    one network engineer who owns the fabric configuration, on-call for it, and builds
           the regression tests: roughly $250,000 per year fully loaded
so         the saving funds about three years of that role at this size, and the decision turns
           on whether that person exists and wants the job
at 128 GPUs the same arithmetic gives about $96,000 of saving, which funds a few months, and
           InfiniBand is the clear answer
sanity: the technology comparison is close enough that staffing and scale decide it, which is
        why two competent teams reach opposite conclusions honestly

The call, for a first training cluster of a few thousand GPUs at an organization without a dedicated fabric team: InfiniBand. The reason is not throughput, since both reach line rate when configured correctly. It is that the failure mode of the alternative is silent, appears only under the congestion that a real job produces, and costs days of a training run to diagnose. Debugging a Slow All-Reduce is the procedure you will be running, and on RoCE it has more branches.

The reversal condition, stated as two concrete triggers rather than a preference. Reverse to RoCE when there is a named network engineer who owns switch configuration as part of their role, with a test that reproduces congestion before a change reaches production, because then the configuration burden has an owner and the per-port saving is real money. Reverse also when the cluster must share a fabric with the rest of the company's infrastructure, because running a second isolated fabric has its own cost and complexity, and at that point the question becomes how to protect the training traffic with quality-of-service classes rather than which fabric to buy. Above roughly ten thousand GPUs the calculation shifts again for a different reason: at that scale you will have fabric specialists regardless, and the per-port saving is large enough to fund a team.

What interviewers probe next

• "What actually breaks first on a misconfigured RoCE fabric?" Either packet loss, which shows as retransmission counters climbing and throughput collapsing, or a PFC pause cascade, which shows as pause frame counters rising fleet-wide while nothing is dropped and everything is slow.
• "How would you validate a RoCE fabric before trusting it?" Run a collective large enough to produce sustained congestion across the whole cluster, not a two-node bandwidth test, and watch pause and ECN counters on the switches while it runs.
• "Does the choice affect the software?" Barely. NCCL uses either through the same verbs interface. RoCE needs the GID index set correctly, which is one more variable to get right.
• "What about Ultra Ethernet?" It is the industry's attempt to remove the configuration burden by redesigning the transport rather than patching Ethernet. It changes this comparison when it is deployable at scale, and until then it belongs in a plan rather than in a purchase.

Common mistakes

• Validating a RoCE fabric with a two-node test, which never creates the congestion the configuration exists to handle.
• Framing the comparison as raw bandwidth, when both reach line rate.
• Choosing RoCE for the capital saving without funding the person who will own the configuration.
• Treating the choice as permanent, when the fabric is the part of a cluster with the longest lifetime and the hardest migration.

Key takeaways

• InfiniBand is lossless by design through credit-based flow control; RoCE is lossless only when PFC and ECN are correct on every hop.
• The RoCE failure mode is silent and appears only under real congestion, which is why two-node validation passes and 512-GPU jobs do not.
• At 1,024 GPUs the per-port saving is roughly $768,000, which funds about three years of the network engineer the choice requires.
• Choose InfiniBand without a fabric owner; reverse when one exists, when the fabric must be shared, or above the scale where you will have specialists anyway.