After recabling a row to 800G, a third of the links will not come up. Find the cause.

TL;DR: Split the symptom first. A link that never trains is a physical or configuration mismatch: wrong cable type for the distance, a speed or mode mismatch between the ends, a transceiver the switch will not accept, or a port administratively down. A link that trains and then flaps is marginal signal integrity, a dirty or damaged connector, or a transceiver running hot. For a fresh recabling to 800G the overwhelmingly likely cause of the first class is reach: passive direct-attach copper reached about 3 m at 400G and tops out near 2 m at 800G, so a bill of materials carried across from the previous generation produces cables that are physically fine and electrically too long. Check ibstat and the switch port state to classify, read the cable's part number and rated length, compare it against the actual run, and only then start swapping parts. Swapping first destroys the pattern that would have identified the cause in one step.

How to approach it

Classify before investigating: never-trained against trained-then-flapping. Then look for a pattern in which links failed, because a third of a row failing is almost never a third of the parts being bad. Read the inventory rather than the cables: part numbers and rated reach against measured run lengths. Then counters. Then, and only then, physical swaps. Close with what you would change in the ordering process.

A strong answer

A typical situation: a row is recabled from 400G to 800G over a weekend using cables ordered from the previous build's part list. On Monday 22 of 64 links are down. The team's first move is to swap transceivers on the failed ports, which fixes nothing and consumes the spares.

Classify first, because the two classes have disjoint causes:

never trained
  ibstat shows the port State: Down, Physical state: Polling or Disabled
  switch port shows no link, no errors, because nothing ever ran
  causes: cable too long for its type, wrong cable class, speed or mode mismatch, an
    unaccepted transceiver, port disabled, or a bent or unseated connector

trained then flapping
  ibstat shows Active, then Down, repeatedly; the switch logs link up and down events
  counters accumulate: symbol errors, link downed counts, effective error counters
  causes: marginal signal integrity, a dirty or damaged fiber end face, a transceiver above
    its temperature rating, or a firmware bug in the link training state machine

why this split matters
  a never-trained link is diagnosed from the inventory and the topology
  a flapping link is diagnosed from counters over time
sanity: 22 of 64 failing at once, immediately after a planned change, with no errors logged,
        is the never-trained class and points at the change rather than at the hardware

The pattern is the diagnosis, and it is free to look for:

group the failed links by anything you can
  by rack position:   are the failures the longer runs?
  by cable part number: is one part number responsible for all of them?
  by switch:          is one switch's ports the whole set?
  by rail:            rail-optimized designs make an entire rail's failure obvious

what a distance pattern looks like
  measured run lengths for the 22 failed ports: 2.5 to 3 m
  measured run lengths for the 42 working ports: under 2 m
  and the part number on all of them is a passive DAC rated at 400G/3 m
sanity: the split falls exactly on a length boundary, which no random hardware failure
        produces, and it identifies the cause without touching a single cable

What the mistake costs, which is the argument for a cable plan:

the failed order
  links in the row                                     64
  links on runs longer than the 800G passive limit     22
  fraction wrong                                       22 / 64 = 34%
  replacement: active copper or optics for those 22
    2 transceivers per link if optical: 22 x 2 = 44 modules
    added power at about 15 W per module: 44 x 15 = 660 W the row did not previously draw

the outage while they are on order
  a rail-optimized design puts one link per GPU, so 22 dead links is 22 GPUs off the fabric
  those GPUs are in nodes of 8, so up to 22 nodes are unusable for multi-node jobs
  22 nodes x 8 GPUs x 24 h x $2.5 = $10,560 per day of idle capacity
  at a two-week lead time on the correct parts: 14 x 10,560 = about $148,000

the plan that would have prevented it
  measure every run, then pick the cable class per run from the rated reach at 800 Gb/s
  cost of doing that: one person for a day
sanity: $148,000 against one person-day, and the day also produces the document that makes
        the next upgrade correct, which is why cable planning is not optional at this scale

Cables, Transceivers and the Optics Power Budget covers the reach ladder. The specific point here is that higher signalling rates lose copper reach every generation, so the same physical cable that carried 400G over 3 m does not carry 800G over 3 m, and this is the single most common ordering mistake in a fabric upgrade.

The commands that produce the evidence:

on the host
  ibstat                       port State, Physical state, Rate, Base lid
  ibstatus                     the same in a more compact form
  ibdiagnet                    fabric-wide: reports links below expected width or speed,
                               and lists ports with errors
  lspci -vv | grep -i mellanox confirms the NIC is present and at the expected PCIe width

on the fabric
  the switch's per-port state, and the transceiver inventory it reports:
    part number, vendor, rated length, and the temperature it is running at
  per-port counters for the flapping class:
    LinkDownedCounter          how many times the link dropped
    SymbolErrorCounter         signal integrity
    PortRcvErrors              received errors
    LinkErrorRecoveryCounter   recoveries, which precede a drop

the physical checks, last
  connector seating: reseat once, and note whether it changed anything
  fiber end faces: inspect and clean before replacing anything
  transceiver temperature from the switch's own reporting, against its rating
sanity: LinkDownedCounter rising while SymbolErrorCounter stays flat points at something
        other than the medium, such as a remote-end reset or a firmware issue, and the two
        counters together are what separates them

NVLink and Fabric Faults covers the counters in more depth, including which values are normal and which are not. The habit worth building is reading counters before swapping parts, because swapping loses the state that would have told you which of these it was.

The reversal condition: if the failures do not correlate with length, part number, switch or rail, and the counters show errors rather than a clean failure to train, then this is not a recabling mistake and the recabling is a coincidence. The candidates then are a firmware version that changed during the same maintenance window, a switch that was also upgraded, or a batch of transceivers with a real defect. The way to tell is that a genuine part defect scatters randomly across positions, and a configuration or specification error falls along a boundary. Look for the boundary first, since it is free, and conclude a part defect only when no boundary exists.

What interviewers probe next

• "Why did the transceiver swaps not help?" Because the fault was the copper cable's rated reach, and the transceivers were never the problem. It also consumed the spares, which matters when the real fix needs parts.
• "What would you order to fix it?" Active copper for the short runs beyond passive reach, or optics with fiber for anything past a few metres, sized from the measured run lengths rather than from the old list.
• "How would you prevent it?" A cable plan derived from measured distances and the rated reach at the target rate, reviewed against the part numbers before the order goes out.
• "What if only one link in a bundle fails?" That is the random pattern, and it points at a single part: reseat, inspect the end face, then replace the transceiver, then the cable.

Common mistakes

• Swapping parts before classifying the symptom, which destroys the pattern and consumes spares.
• Reusing the previous generation's cable lengths at a higher signalling rate.
• Treating never-trained and flapping links as one problem, when their causes do not overlap.
• Ignoring the run-length distribution, which is usually the whole diagnosis and costs nothing to check.
• Concluding a bad batch of parts before checking whether the failures fall on a boundary.

Key takeaways

• Classify first: never-trained points at cable class, reach or configuration; flapping points at signal integrity, connectors or heat.
• Passive copper reached about 3 m at 400G and about 2 m at 800G, so reused part numbers fail on the longer runs.
• Group failures by run length, part number, switch and rail; a clean boundary identifies the cause without touching hardware.
• Read ibstat, ibdiagnet and the per-port LinkDownedCounter and SymbolErrorCounter before swapping anything.
• A random scatter of failures across positions is a part defect; a boundary is a specification error.