TL;DR: This question tests whether you treat researchers as customers or as adversaries, and the wrong answer is visible immediately. The working approach has three moves. Replace opinion with data by showing the queue: what is running, what it is worth, and when the next 256-GPU window opens, because most of these arguments are between two people who each believe the cluster is being wasted on the other. Offer something today that is smaller than the ask, since a researcher who wanted 256 GPUs for a week can often start on 32 while the shape of the experiment is still wrong, and starting is worth more to them than waiting. And give them a lever they control, meaning a quota they can spend and preempt within, so the next conversation happens inside their own budget instead of in your inbox. Underneath all three is the same principle: the scarce resource is not GPUs, it is the researcher's willingness to plan, and you buy that with predictability rather than with rules.
How to approach it
Ask what the experiment is before answering, because the answer changes completely between a debugging run and a scheduled sweep. Then say what you can offer today. Then say when the full ask is available and how confident that date is. Then, separately from this instance, name the mechanism that stops it recurring. Interviewers are listening for whether you jump to the mechanism and skip the person, which is a common failure in platform engineers.
A strong answer
A typical situation: a researcher messages at 10 a.m. asking for 256 GPUs to start a run before a paper deadline. The cluster is at 94% allocation, the queue has four jobs ahead, and one of the running jobs belongs to a team that reserved capacity three weeks ago and is using 40% of it.
The first move is to get the real request, which is rarely the stated one:
| What they say | What is often true | What you can offer |
|---|---|---|
| "I need 256 GPUs today" | The first two days are debugging at small scale | 16 to 32 GPUs immediately, full allocation on Thursday |
| "It has to be one job" | It has to be one result, and the job could be four | Four 64-GPU jobs starting now, if the code supports it |
| "It cannot be preempted" | It cannot lose more than an hour of progress | Preemptible with checkpointing every 20 minutes |
| "I need it for a week" | Nobody knows; the estimate is a guess | The window, plus a check-in at day three |
None of these are negotiating tactics. They are the questions that reveal what the researcher actually needs, and asking them is what makes the eventual no credible when it is a no. Working with Researchers covers this relationship in more detail, including why a fast path for small jobs pays back more than any other thing a platform team can build.
The second move is showing the queue rather than describing it:
what to put on the screen
running jobs with owner, size, start time and current GPU utilization
the queue with position and estimated start
the next window large enough for the ask, and how it was computed
why this works
the argument "the cluster is full" is an assertion the researcher cannot check, and
people do not accept assertions about resources they need
the same sentence with the queue on screen becomes a shared fact, and the conversation
moves to what to do about it
it also exposes the real problem when there is one: a reservation running at 40%
utilization is a conversation with that team, and it is a conversation you can now have
with a number instead of a suspicion
The third move is the one that ends the weekly recurrence. Give each team a quota they own, with the ability to preempt their own jobs, and let borrowing across teams happen automatically when capacity is idle. Multi-Tenancy, Quotas and Fair Share covers the mechanics, and the behavioral point is narrower: once a researcher can preempt their own low-priority job to start a high-priority one, the request stops arriving in your inbox and starts being a decision they make. That is a better outcome for both people, and it is the answer that separates a platform engineer from a ticket queue.
The reversal condition: sometimes taking capacity from a running job is correct, and refusing to ever do it is its own failure. When a deadline is real and organizationally agreed, and the running job is a low-priority sweep whose owner can restart from a checkpoint, preempting it is correct. What makes that survivable is that the policy existed before the incident: a published priority band, a checkpoint requirement for jobs in the preemptible tier, and a notification that arrives before the preemption rather than after. Doing it without that policy, as a one-off favor, is the decision that destroys the queue's credibility for the next six months, because every team then learns that asking loudly works.
What interviewers probe next
- "What if they escalate to your manager?" Say what you would want your manager to see: the queue, the offer you made, and the date. An escalation that arrives with your data already attached is not a threat.
- "How do you decide priority between two researchers?" Not by who asked louder. By an agreed banding, usually tied to what the organization has committed to, and the honest answer names who owns that decision when two teams cannot agree.
- "What if the cluster really is being wasted?" Then that is the finding, and it is worth more than this one allocation. Bring the utilization data to the team that owns the reservation, privately first.
- "How do you keep researchers from padding their requests?" Make the small allocation instant and the large one scheduled. People pad when waiting is the only failure mode.
Common mistakes
- Answering with policy immediately, which reads as someone who does not want to understand the request.
- Saying no without an alternative, when a smaller allocation today is almost always available.
- Making an exception without a policy behind it, which teaches everyone that escalation works.
- Describing the queue instead of showing it, so the constraint stays an assertion.
- Treating a low-utilization reservation as the researcher's problem rather than as your finding.
Key takeaways
- Ask what the experiment is first: a debugging run and a scheduled sweep are different requests wearing the same words.
- Offer a smaller allocation now plus a dated window for the full ask, because starting beats waiting for most experiments.
- Put the queue on screen; "the cluster is full" is an assertion, and a queue with positions and start times is a shared fact.
- Give teams a quota they can preempt within, which moves the decision from your inbox to theirs.
- Preempt a running job only under a policy that existed beforehand, with notification and a checkpoint requirement.