TL;DR: The question is not asking for drama. It is asking whether you can operate a system you do not fully understand while it is broken and people are waiting. Five beats carry that. The signal, meaning what you looked at first and why that rather than something else. The hypothesis, meaning what you thought was happening and how you tested it cheaply. The decision, meaning what you did before you were certain, which is the beat the whole question exists to reach. The recovery, with a time. And the change, meaning the alert, the runbook or the design fix that means the next person does not repeat your night. Give the timeline in wall-clock terms and keep the total under three minutes. If the root cause turned out to be a vendor's firmware or another team's deploy, tell it that way and spend the story on your response rather than on their mistake, because the interviewer is evaluating you and not them.
How to approach it
Set the scene in two sentences: what was broken, who was affected, and what the clock was. Then walk the timeline forward, marking each point where you had to choose without knowing enough. Say the wrong turn you took, because every real incident has one and a story without one sounds rehearsed. End on the change and how you know it worked.
A strong answer
A typical situation: a training job on 512 GPUs starts failing to restart after a normal preemption, and the queue behind it backs up. Six research teams cannot start work. It is 2 a.m., the engineer on call did not build the checkpoint loader, and the error is a timeout with no stack.
The five beats, with what each one is actually testing:
| Beat | What you say | What is being scored |
|---|---|---|
| Signal | "I looked at whether any rank had progressed, not at GPU utilization, because a stuck collective shows full utilization." | Whether you know which metric lies. |
| Hypothesis | "Restarts had worked at 12:40 and failed at 1:05, so I diffed what changed in that window: a storage maintenance event on the checkpoint tier." | Whether you narrow by time before you narrow by component. |
| Decision | "I could not confirm the cause, so I pointed restarts at the previous checkpoint on the replica tier, losing 40 minutes of training across six jobs, and I said so in the channel before doing it." | Whether you act under uncertainty and account for the cost out loud. |
| Recovery | "Jobs were running again 22 minutes later. Root cause came the next morning from the storage team's event log." | Whether you separate mitigation from root cause. |
| Change | "The checkpoint loader now fails fast with the path and the error instead of timing out, and there is an alert on restart failures per queue rather than per job." | Whether anything outlasted the night. |
The decision beat is where most tellings collapse. Candidates describe investigating until they found the cause and then fixing it, which is what happens in small incidents and almost never in large ones. Large incidents are resolved by mitigating before the cause is known, and the interviewer wants to hear you weigh a cost you cannot fully estimate. Naming the price of the mitigation, 40 minutes of training across six jobs, and saying you announced it before acting, is the specific thing that reads as senior. Incident Response for GPU Fleets covers the mitigation-first structure and the roles that make it work at fleet scale.
Two framings to avoid, and what to say instead:
"the vendor's firmware was the root cause"
true, possibly, and it is not your story. Spend one sentence on the cause and the rest on
how you found it, what you did before you knew, and what you changed so the next firmware
regression is caught by a canary rather than by a customer.
"we were up in eight minutes"
a fast recovery is good and it makes for a thin story. If the only incident you can
describe was resolved in eight minutes by a runbook, pick a different one. The question
wants the case where the runbook did not cover it.
On-Call Narratives That Land goes through the structure and the way to compress a six-hour night into three minutes without losing the decisions. The story also gets much stronger when you can say what the incident cost in terms the business recognizes: GPU-hours lost, customers affected, or hours of researcher time, because that is the unit the follow-up questions are asked in.
The reversal condition: when the incident was one where you were not the primary responder, do not inflate your role. Interviewers ask "what did you personally do" precisely to catch this, and a story where you ran comms, kept the timeline, and freed the primary to debug is a strong story told honestly. Claiming the debugging and then failing the follow-up on a detail is the single fastest way to lose a behavioral round.
What interviewers probe next
- "What did you look at first, and why not something else?" This is the whole question in miniature. Have the reason, not just the metric.
- "What did the mitigation cost?" Name it in GPU-hours, dollars or researcher-hours. An engineer who mitigated without knowing the cost is a different answer from one who priced it and chose.
- "What did the postmortem change?" Give one concrete artifact and say whether it has fired since. "We added an alert" is weaker than "we added an alert and it caught the same class of failure in March."
- "Who did you tell, and when?" Communication during an incident is scored separately from the fix, and candidates routinely omit it entirely.
Common mistakes
- Telling the incident as a mystery with a clever ending, which hides the decisions the question is about.
- Skipping the mitigation and describing only the root-cause investigation.
- Omitting communication, so the story has no one else in it.
- Choosing an incident that a runbook already covered, which leaves nothing to evaluate.
- Overstating your role and then contradicting yourself under a detail question.
Key takeaways
- Five beats: signal, hypothesis, decision under uncertainty, recovery with a time, and the change that outlasted it.
- Mitigate before the cause is known, name what the mitigation costs, and say you announced it first.
- Separate mitigation time from root-cause time; they are different numbers and conflating them reads as imprecision.
- If the cause was a vendor or another team, spend one sentence there and the rest on your response.
- Pick an incident the runbook did not cover, and be honest about which parts you personally ran.