Describe a time you were confidently wrong about a root cause. What did it cost?

TL;DR: The mistake almost always has the same shape: you anchored on the last thing that broke, or on the most recent change, and then read ambiguous evidence as confirmation. In infrastructure this is especially common because so many symptoms are shared. A hang, a slowdown and a crash each have a dozen possible causes and the observable difference between them is often one counter nobody checked. Tell the story with three things in it. The specific bias, named, since "it was a tricky bug" is not an answer about you. The cost, in hours of your time, capacity idled or an outage extended, because a mistake with no stated cost sounds like it did not matter. And what you do differently now, which for most people is writing down two or three hypotheses and the observation that would distinguish them before touching anything. The strongest version includes the moment you noticed you were wrong and what made you notice, since that is the skill actually being hired for.

How to approach it

Give the symptom and your first theory in two sentences. Say why the theory was reasonable, because a story where you believed something stupid teaches nothing. Then the evidence you read as confirming and what it actually meant. Then the cost. Then how you found out, and finally the change. Do not soften the ending; a candidate who says the wrong theory cost the team six hours is more credible than one whose mistake conveniently cost nothing.

A strong answer

A typical situation: a training job on 512 GPUs slows from 3.1 seconds per step to 4.4 over a week. The previous month a similar slowdown was caused by a degraded NVLink on one node, so the engineer starts there, finds a node with a slightly raised replay count, drains it, and the slowdown continues. Two days later someone notices that the checkpoint interval had been reduced from 30 minutes to 10 in a config change, and the step time average now includes checkpoint stalls.

The bias by name is anchoring, and the confirming evidence was the replay count, which was above baseline and was not the cause. That combination is the classic version of this mistake:

The distinguishing observation existed the whole time and was cheap. A step-time histogram would have shown a bimodal distribution, with most steps at 3.1 seconds and every tenth step long, which is a checkpoint signature and not a straggler signature. A straggler produces a uniformly slower distribution. Stragglers and Hangs covers that distinction and the per-rank timing that separates the two in one plot.

The cost, stated plainly, is what makes the story land:

what the wrong theory cost
  the drained node: 8 GPUs out for 2 days = 384 GPU-hours
  at $2.5 per GPU-hour that is about $960 of capacity
  the engineer's time: roughly 6 hours across two days
  the slowdown itself continued for those 2 days: 512 GPUs running 42% slower than they
    should have, which dwarfs both of the above

why saying this matters in the interview
  a candidate who cannot price their own mistake has probably never had to explain one to
  someone who cared about the number
  and the largest cost here is the one that is easiest to forget, which is the time the
  actual problem kept running

The process change is the part that turns this into a lesson. The version that works is small and specific: before acting, write down two or three hypotheses and, for each, the observation that would rule it out. It takes four minutes, it makes anchoring visible because the list forces at least one alternative, and it produces a record that helps whoever picks the incident up next. On-Call Narratives That Land covers how to keep that record while the incident is live without slowing the response.

The reversal condition: this discipline can be overdone, and an engineer who writes hypothesis lists during a customer-facing outage is optimizing the wrong thing. When the system is down and a known mitigation exists, apply the mitigation and investigate afterward. The hypothesis discipline belongs to the slow class of problems, the ones that have been degrading for a week and where the cost of a wrong guess is days rather than minutes. Knowing which class you are in is itself part of the answer.

What interviewers probe next

• "What made you finally notice?" Name it. Someone else's question, a metric you had not looked at, or the fix not working. The last one is the most common and is fine to say.
• "How long did you hold the wrong theory?" Have the number. Two days is a normal answer; four hours suggests a small incident, and two weeks invites a follow-up about why nobody challenged it.
• "What would have caught it earlier?" The cheap distinguishing observation. If you cannot name one, the story is incomplete.
• "Has the change stuck?" Say honestly whether you still do it. A process change you abandoned after a month is a more interesting answer than a claimed habit.

Common mistakes

• Choosing a story where being wrong cost nothing, which removes the point of the question.
• Blaming the evidence for being misleading instead of naming your own reasoning error.
• Saying "I learned to be more careful", which is not a process change and cannot be checked.
• Omitting how you found out, so the story has no correction mechanism in it.
• Picking a technically trivial mistake to look safe, which reads as an unwillingness to be examined.

Key takeaways

• Anchoring on the last incident plus confirming evidence with an unknown base rate is the standard shape of this mistake.
• Price the mistake: idled capacity, your hours, and the time the real problem kept running, which is usually the largest term.
• The cheap distinguishing observation usually exists; a bimodal step-time histogram is a checkpoint signature and a uniformly slower one is a straggler.
• The process change that works is two or three written hypotheses with a ruling-out observation each, taking about four minutes.
• Skip the discipline during a live outage with a known mitigation, and say that you would.