Each rank keeps a ring buffer of its recent collectives, and on a timeout every rank dumps it. Comparing sequence numbers across a thousand of those buffers finds the missing participant in one pass, and the three patterns those comparisons produce point at three different causes.
Walk me through reading a collective flight-recorder dump. What is in it, and how do you find the rank that caused a hang?
Each rank keeps a ring buffer of its recent collectives, and on a timeout every rank dumps it. Comparing sequence numbers across a thousand of those buffers finds the missing participant in one pass, and the three patterns those comparisons produce point at three different causes.
Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.
The concepts behind this question
Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.
Scored on the record's fields and what each is for, on the comparison across ranks rather than reading one buffer, and on the three patterns (one behind, mismatched operation, all equal) mapping to distinct causes.
No comments yet — be the first to share your approach.
