The Reliability Pushback Story
Every AI infra loop has a behavioral round, and the story it wants most is the one where you stopped something (a launch, a run, a hardware admission) because the data said to, and you were accountable for the cost of stopping. This page gives the skeleton that works: the situation, the signal you read, the decision and who owned it, the evidence you brought, and what changed afterward. It also gives the follow-up interviewers hold back, the version that sounds right and fails, and the line between a senior telling and a staff telling of the same story.
TL;DR: Tell it in five beats: the situation with its stakes in GPU-hours or user minutes, the specific signal you read (a counter, a burn-in result, an error budget), the decision you made and who had the authority, the evidence you put in front of the person who disagreed, and the thing that changed so the same argument never had to happen again. State what stopping cost, out loud, before the interviewer asks. The version that fails is the one where you were right, the cost was zero, and nothing changed afterward.
What the round is testing
Every loop in the research has a round for this. OpenAI's onsite carries a 45-minute behavioral with a senior manager and a 30-minute teamwork behavioral. Anthropic runs an hour-long values and culture round, which recruiters report as the round where most rejections happen. Amazon scores Leadership Principles with a bar raiser in the room. Google's committee scores leadership alongside coding. NVIDIA and AMD put a behavioral with an engineering manager into loops that are otherwise pure domain depth. The prompt varies ("tell me about a time you disagreed with a decision", "a time you stopped something", "a time you were the only one who saw a problem"), and the scoring does not: did this person read a real signal, act on it with a cost attached, and leave the system better than the argument found it.
Infrastructure engineers have an advantage here that most of them waste. The job produces these stories every quarter. A delivery of nodes that wants to join a run before burn-in is done. A model launch that wants to skip the canary because the deadline is Thursday. A researcher who wants checkpointing turned off for a week to buy 3% step time. The candidate who has lived any of those and tells it flat, with numbers, beats the candidate who tells a polished story about a code review.
The skeleton
| Beat | What it contains | The sentence it produces |
|---|---|---|
| situation | the system, its size, who depended on it, the deadline | "A 2,048-GPU pretraining run, three weeks from a milestone, and 64 new nodes arriving to join it" |
| the signal | one concrete measurement you read, and why it alarmed you | "The burn-in pass rate on the new batch was well below what the previous batch had shown, and two nodes had already logged uncorrectable ECC errors" |
| the decision | what you said no to, to whom, and what you offered instead | "I told the run owner we would not admit the batch that day, and offered a 48-hour extended burn-in with admission of whatever passed" |
| the evidence | the arithmetic or the log you put on the table | "At the failure rate the burn-in implied, the run would restart every few hours instead of every day, which cost more step time than waiting" |
| what changed | the gate, alert, or policy that made the argument unnecessary next time | "Admission now requires a burn-in pass, and the run owner reads the same dashboard I do" |
The beats are in that order because the interviewer's questions arrive in that order. Skip the signal and the story is an opinion. Skip the decision's owner and the story is a complaint. Skip what changed and the story is an anecdote.
A worked telling: the node admission
The situation is a typical one. A frontier run is short of its target size, a hardware batch lands, and the scheduler is ready to admit it. The person who owns the run wants the nodes today. You own fleet health.
The signal is the part most candidates blur. Say what you read. The burn-in suite on the batch reported a pass rate visibly below the previous delivery's. Two GPUs had already shown row-remapping events in their first day. The NVLink counters on a handful of nodes were climbing during the fabric test. None of that is a proof; it is a pattern the previous batch did not have (Node Health Checks and Burn-In).
The decision names the authority. "I could block admission; the run owner could escalate over me. I said we would not admit the batch that day, and I put the alternative next to the refusal: a longer burn-in, admit whatever passed, and I would personally sit the extended run." The alternative matters. A no without an alternative reads as obstruction; a no with a dated alternative reads as ownership.
The evidence is one chain, said in two sentences. The run at its current size was restarting roughly once a day, which the team could live with. If the new batch failed at the rate its burn-in implied, restarts would come several times a day, and every restart cost the work since the last checkpoint plus the restart time itself (Training Interruption Statistics). Two days of waiting cost less step time than one week of that.
What changed: admission gained a burn-in gate, the run owner got the same fleet dashboard, and the escalation path was written down so the next disagreement went to a person with a name rather than to whoever argued longest.
Then the cost, unprompted. "Waiting cost the run two days of the extra capacity. If I had been wrong about the failure rate, that was two days of a 64-node contribution for nothing, and I would have owned that." Interviewers listen for this sentence. Most candidates never say it.
A second telling: the launch canary
A serving story has the same shape with different nouns. A new model version is due at a date the product team announced. The launch plan skips the canary because the previous version's canary "never found anything." Your signal is the load test: p99 time to first token at the announced traffic sits above the SLO on the new engine build, and the number is reproducible (SLOs for AI Systems). The decision: ship on the date, behind a flag, at 5% of traffic, with the old version taking the rest, and a written rollback trigger on the error budget burn rate. The evidence: the p99 curve from the load test next to the SLO line. What changed: the canary is now the default for engine builds, and the rollback trigger is an alert rather than a judgment call at midnight. The cost: the full launch landed a week later than the announcement, and you owned the conversation with the product team.
The follow-up they hold back
"What if you had been wrong?" is the one to prepare for. A weak answer defends the decision again. A strong answer names the price of being wrong (two days of capacity; a week of launch delay), says you knew that price when you decided, and states the number that would have changed your mind. That last part is the real test: a candidate with a reversal condition made a decision; a candidate without one had a feeling.
The second held-back follow-up is "who made the final call?" Say it plainly. If it was your call, own it. If it went above you and they sided with you, say so and describe what you gave them to decide with. If they overruled you, the story is still good, provided what changed afterward is real.
The answer that sounds right and fails
It goes like this: everyone wanted to ship, you alone saw the risk, you argued hard, they listened, disaster was averted. It fails on four counts. No signal was named, so the interviewer cannot tell whether you read data or had a hunch. No cost was stated, so the decision looks free. The other side is a straw figure, so there was no real trade to weigh. Nothing changed, so the same fight is due next quarter. The same story with the burn-in number, the two-day cost, a run owner who had a legitimate deadline, and the gate that followed is a different story in the interviewer's notes.
A related failure is the story where the pushback was "I raised it in the meeting" and the decision belonged to someone else. That is a story about noticing, which is fine for a mid-level loop and thin for a senior one.
Senior and staff tellings
The senior version stops the launch with data and owns the cost. The staff version does that and also changes the shape of the decision: the gate that makes the next batch admit itself, the dashboard both sides read, the written escalation path. Staff candidates also tend to name the trade honestly in the other direction, the times they let something through under a known risk because the expected loss was smaller than the delay. An interviewer who hears only refusals is hearing a gatekeeper. One who hears refusals and calculated admissions is hearing judgment (Leveling Signals: Senior vs Staff).
Working it in the room
Pick the story before the loop, one from training and one from serving, and rehearse each to under three minutes. Open with the situation and its size in one sentence. Put the signal in the second sentence with a number in it. State the decision and the alternative you offered together. Give the evidence as one chain. Close with what changed and the cost, before being asked. Then stop talking; the follow-ups are where the round is scored, and a candidate who leaves room for them looks more confident than one who fills the time.
What to remember
- Five beats: situation, signal, decision and its owner, evidence, what changed. In that order.
- The signal is a measurement (a burn-in rate, a counter, a p99 against an SLO), never a feeling.
- Every no comes with a dated alternative.
- Say the cost of stopping before they ask, and the number that would have changed your mind.
- Senior stops the launch; staff also removes the need for the next argument.