You are joining a team running a 10,000 GPU cluster. What do you do in your first month?

TL;DR: Four things, in order. Read the last year of postmortems, which is the highest-density source of truth about any cluster because it describes what actually breaks rather than what the design intended. Draw the map by following one training job end to end, from submission through scheduling, image pull, storage mount, fabric assignment, the health check that gates it, and the checkpoint path, because that traversal touches every subsystem and shows you where the boundaries between them are. Shadow on-call for at least one rotation, since the fastest way to learn which alerts matter is to watch someone triage them. And ship one small fix, ideally something that annoyed you in week one, because it teaches you the deploy path and buys credibility that a month of asking questions does not. End the month with a written list of questions nobody could answer, which is more valuable to the team than anything else you produce and is often the thing that gets remembered.

How to approach it

Answer in weeks, because the question is about sequencing. Say what you are learning in each phase and what artifact comes out of it. Be explicit that you are not proposing architecture changes in the first month and say why. If the interviewer pushes on why you would not fix the obvious problem you noticed on day three, that is the real question, and the answer is that you do not yet know why it is that way.

A strong answer

A typical situation: an engineer joins a team running a 10,000 GPU cluster with 40 training teams on it. There is a wiki that is 18 months stale, a Slack channel with everything in it, and nobody with time to onboard them properly. The team's real question is whether this person will be net positive by month three.

The four weeks, with what each produces:

Postmortems are the highest-value hour in that table and are usually skipped. They tell you what breaks, how the team talks about failure, whether the culture is blameless in practice as well as in the template, and which parts of the system nobody understands. Five failure classes typically cover most of what a large cluster does to itself, and Training Cluster Design at 10k GPUs covers what those classes are in a well-run fleet, which gives you a baseline to compare against.

The traversal in week two is the one that builds the mental model:

follow one job and write down what you learn at each hop
  submission        who can submit, what the quota check does, where it is recorded
  scheduling        gang or not, what happens when the gang cannot be placed
  image pull        where images come from, how long a cold pull takes, is it cached
  storage           what gets mounted, from where, how the credentials arrive
  fabric            how ranks are assigned to topology, whether it is topology-aware
  health gate       what is checked before a node joins a job, and what is not checked
  the run           what the step time is, what MFU it achieves, who watches it
  checkpoint        interval, destination, how long a write takes, how a restart finds it
  failure           what happens when one rank dies, and how long the restart takes

what this produces beyond the diagram
  a list of the numbers this cluster runs at, which you will need for every conversation
  the places where the answer was "I do not know" or "ask that team"

Two of those hops repay extra attention. The health gate is where Node Health Checks and Burn-In describes what a well-run fleet screens for, and comparing that list against what this cluster actually checks usually finds a gap in week two. The step time and MFU line is where MFU and HFU gives you the number to compare against, and a cluster running well below its class is a fact worth knowing before you have opinions about anything else.

Those "I do not know" answers are the question list, and it is the artifact worth the most. A new engineer is the only person who can produce it, because it stops being visible after about six weeks. Write it down, share it, and do not present it as criticism.

The reversal condition: if the team is in the middle of an incident or a crunch when you join, this plan is the wrong one and insisting on it reads badly. The right move then is to make yourself useful at whatever level you can, which usually means taking the low-context work off someone so they can focus, and doing the learning inside that work. The plan above assumes a normal month, and saying that assumption out loud, then giving the alternative, is a stronger answer than the plan alone.

What interviewers probe next

• "What would you fix first?" Name something small and real, and say explicitly that you would not touch the scheduler in month one because you do not yet know why it is that way.
• "What if the postmortems do not exist?" Then that is a finding, and the substitute is the incident channel and the oldest engineer's memory. Say you would start writing them.
• "How do you learn without slowing the team down?" Batch questions, ask them in one session per person, and write the answers somewhere shared so nobody answers twice.
• "What if the wiki is wrong?" Assume it is. The traversal in week two exists precisely because documentation drifts and a job's actual path does not.

Common mistakes

• Proposing architectural changes in the first weeks, before knowing the constraints that produced the current design.
• Skipping postmortems, which is the densest available source of how the system really behaves.
• Shadowing on-call while trying to help, which slows the responder and teaches you less than watching does.
• Producing a document instead of shipping something, so month one ends with no evidence you can operate the system.
• Presenting the question list as a critique rather than as the artifact only a newcomer can make.

Key takeaways

• Order: postmortems, then the end-to-end job traversal, then shadowing on-call, then one shipped fix.
• The traversal touches submission, scheduling, image, storage, fabric, health gate, run, checkpoint and failure, and gives you the cluster's numbers.
• Shadow a full rotation without touching anything, and count how many alerts were actionable.
• Ship something small in week four to learn the deploy path, not to prove a point.
• The list of questions nobody could answer is the artifact only a new person can produce, and it expires in about six weeks.