TL;DR: Incast is many senders transmitting to one receiver at once, which happens naturally in collectives and in storage reads. At 400 Gb/s a sender delivers 50 GB per second, so 32 senders aimed at one port deliver 1.6 TB per second into an egress port that can drain 50 GB per second, and a shared buffer of a few tens of megabytes fills in tens of microseconds. Two mechanisms respond. Explicit congestion notification marks packets once the queue crosses a threshold, the receiver reflects the mark, and the sender reduces its rate: this is the slow, well-behaved path, and it should be the one that fires. Priority flow control sends a pause frame to the upstream device, which stops transmitting entirely: it prevents loss but propagates backward, and a chain of pauses can stall an entire fabric including traffic unrelated to the congestion. The design goal is to set the ECN threshold low enough that senders slow before the buffer reaches the pause threshold, so PFC remains a safety net rather than a control loop.
How to approach it
Establish the timescale first, because it explains why this cannot be handled in software. Then describe both mechanisms and, more importantly, the order they are supposed to fire in, which is where configurations go wrong. Then describe the failure that follows when the order is inverted. Close with the counters, because the two conditions look identical from a job's point of view and different from the switch's.
A strong answer
A typical situation: a cluster runs well until a checkpoint write, when unrelated training jobs on other racks slow down for the duration. Nothing is dropping packets and no error appears in any log. The storage traffic is triggering pause frames on a shared uplink, and the pauses are propagating to ports carrying other jobs' collectives.
The timescale, which is why this is a hardware mechanism:
sender rate 400 Gb/s = 50 GB/s per port
incast fan-in 32 senders to one receiver (the last step of a reduce-scatter, or 32
clients reading one storage node)
arrival rate 32 x 50 = 1,600 GB/s into an egress port that drains at 50 GB/s
excess 1,550 GB/s accumulating in the switch buffer
switch buffer a modern data-center switch has tens of megabytes shared across ports;
take 64 MB and assume this flow may use a quarter of it, 16 MB
time to fill 16 MB / 1,550 GB/s = about 10 microseconds
sanity: 10 microseconds is far below any software reaction time, and roughly the round-trip
time across the fabric, so the control loop has barely one round trip to act. That is
why the response is built into the switches and NICs rather than into the application
The two mechanisms, and the order they should fire in:
explicit congestion notification (ECN), the one that should act
the switch marks a bit in packets once the egress queue passes a threshold
the receiver reflects the mark back to the sender in a congestion notification
the sender's algorithm (DCQCN on RoCE) reduces its rate, then probes back up
properties: end to end, per flow, gradual, and it slows only the senders responsible
priority flow control (PFC), the last resort
the switch sends a pause frame to the upstream device on that priority class
the upstream stops transmitting on that class entirely until released
properties: hop by hop, per class rather than per flow, immediate, and it stops all traffic
in that class from that device including flows that had nothing to do with the congestion
the intended relationship: the ECN threshold sits well below the PFC threshold, so rates fall
before any pause is needed. PFC then only fires for a burst too fast for the control loop
The cascade is the failure to be able to describe, because it is what makes a misconfigured RoCE fabric worse than a lossy one. A pause stops the upstream device, whose own buffer then fills, so it pauses its upstream in turn. The stall spreads backward along every path feeding the congested point, and because pauses act on a priority class rather than on a flow, traffic that never touched the congested port is stopped too. A fabric in this state shows high pause counters everywhere, near-zero drops, and terrible throughput, and it looks from inside a job exactly like a slow network.
Congestion Control for AI Fabrics has the mechanisms in full; RDMA, InfiniBand and RoCEv2 covers why loss is not an option in the first place, since RDMA's retransmission behavior at these rates makes even light loss catastrophic for throughput.
The counters that distinguish the states:
healthy under load
ECN marked packets: non-zero and steady
PFC pause frames: near zero
discards: zero
throughput: near line rate
ECN not firing early enough (thresholds too high, or ECN not enabled)
ECN marks: low or zero
PFC pauses: climbing on the congested port and its upstreams
discards: zero
throughput: collapsed, and unrelated jobs affected
PFC not configured at all on a RoCE fabric
discards: climbing
retransmissions: climbing on the NIC counters
throughput: collapsed for the affected flows
sanity: the three states are distinguishable in one glance at two counters, marks and pauses,
which is why those two belong on the fabric dashboard rather than in an investigation
What a design does about incast beyond the mechanisms: spread the fan-in so it does not concentrate, which is why storage is striped across many nodes rather than served from one; use adaptive routing or packet spraying so a burst uses many paths rather than one; and give storage traffic its own priority class so a checkpoint burst cannot pause the class carrying collectives, which is the direct fix for the opening scenario.
The reversal condition: on InfiniBand this discussion changes shape. Credit-based flow control means a sender never transmits without buffer space, so the loss question does not arise and there is no pause cascade of the RoCE kind, though congestion still causes queueing and the fabric has its own congestion notification. The counters and the tuning differ, and a team moving from one fabric to the other should not carry over the thresholds or the dashboards.
What interviewers probe next
- "Why is a pause worse than a drop for unrelated traffic?" A drop affects one flow. A pause stops a whole priority class on a link, including flows that were not congesting anything.
- "How would you set the ECN threshold?" Low enough that marking begins well before the pause threshold, and high enough that normal bursts do not throttle senders. It is tuned per switch model against a real collective, not copied between fabrics.
- "What is the role of the priority class?" It separates traffic that can pause from traffic that must not. Storage and collectives in different classes means one cannot stall the other.
- "Does adaptive routing remove the need for this?" It reduces hot spots by spreading flows across paths, which makes incast less likely at a given load, and it does not remove the case where many senders target one receiver by construction.
Common mistakes
- Treating pause frames as a normal part of operation, when they mean the ECN loop failed to act in time.
- Tuning thresholds by copying values from another switch model or another cluster.
- Putting storage and collective traffic in the same priority class, so a checkpoint burst can stall training.
- Diagnosing "the network is slow" from inside a job without ever reading the switch counters, where the two possible causes look completely different.
Key takeaways
- Incast fills a switch buffer in about 10 microseconds at 400 Gb/s with 32 senders, which is why the response is in hardware.
- ECN marks and senders slow down: gradual, per flow, and the mechanism that should fire.
- PFC pauses the upstream: immediate, per class, and it propagates backward into a cascade that stalls unrelated traffic.
- Two counters tell you which state you are in: ECN marks steady with pauses near zero is healthy; pauses climbing with no drops is the cascade.