What is gang scheduling, and what goes wrong on a Kubernetes cluster that does not have it?

TL;DR: Gang scheduling admits a multi-pod job only when every pod can be placed, binds them together, and tears the job down together if a member fails. Without it, the default scheduler places pods one at a time, so two 40-pod jobs on a 64-GPU cluster can each get 32 GPUs, block at the first collective waiting for ranks that will never arrive, and hold the whole cluster at zero useful work until a human deletes one. Kueue does this at admission, holding the job suspended until quota and capacity allow all of it; Volcano and KAI replace the scheduler with one that understands PodGroups.

How to approach it

Define it in one sentence: the job, not the pod, is the unit of scheduling. Then show why training needs it: every rank blocks on the first collective, so a partial job holds GPUs and does nothing. Count the deadlock on a small cluster so it is concrete. Name the implementations and where each sits. Close with the two failure modes gang scheduling introduces (fragmentation and starvation of large gangs) so the interviewer hears you have run one.

A strong answer

A typical situation: a team submits two training jobs to a new Kubernetes cluster on a Friday, the dashboards show every GPU allocated all weekend, and on Monday neither job has logged a single step. Nothing crashed. The pods are Running, the GPUs are 0% busy, and each job's log ends at torch.distributed.init_process_group waiting for the world to form.

The mechanism: a data-parallel training job with 40 ranks calls a collective at its first step, and that collective returns only when all 40 ranks have joined. A rank that is scheduled runs to that call and waits. A rank that is Pending never joins. The scheduled ranks hold their GPUs and wait forever, and the Kubernetes scheduler, which succeeded at placing each pod it placed, sees nothing wrong.

cluster: 8 nodes × 8 GPUs = 64 GPUs, default scheduler, no gang scheduling
job A: 40 pods × 1 GPU;  job B: 40 pods × 1 GPU, submitted seconds apart
  pods arrive interleaved; the scheduler places each one it can
  A gets 32 placed, 8 Pending;  B gets 32 placed, 8 Pending;  64 of 64 allocated
  A's 32 ranks block waiting for 8 more; B's 32 block waiting for 8 more
  neither finishes, neither releases, nothing new can start
useful work = 0 ÷ 64 = 0%;  reported allocation = 64 ÷ 64 = 100%
with gang scheduling:
  A is admitted when 40 are free: A runs on 40, 24 free
  B needs 40 > 24: B waits in the queue holding nothing
  A finishes; B is admitted. Useful work: 100% of what runs, minus B's wait.
sanity: the failure is a live-lock, and every dashboard says the cluster is full;
        "add nodes" makes it worse because the new nodes fill the same way

Gang scheduling makes the job the unit. The job declares its gang: a Workload in Kueue, a PodGroup with minMember in Volcano or KAI. The scheduler simulates placing all members; if they fit, it binds all; if not, it binds none and the job stays queued. When a member dies, policy decides whether the gang is torn down and requeued, which for training is the right default because a lost rank means a restart from checkpoint anyway.

Where the implementations sit matters for the follow-ups:

• Kueue is an admission layer in front of the default scheduler. Jobs are created suspended; Kueue checks the team's quota and whether the cluster can hold the gang, then unsuspends the job so the default scheduler places all its pods at once. Every other Kubernetes feature keeps working, and Kueue models teams as cohorts of queues with borrowing.
• Volcano is a replacement scheduler with batch plugins (gang, priority, DRF, binpack) shipped together; it is common on on-premises AI platforms.
• KAI Scheduler, open-sourced from Run:ai in 2025, is a replacement scheduler built for GPU fleets with fractional GPUs and many small tenants.

The Gang Scheduling with Kueue and Volcano concept page has the comparison table; in the room, say "Kueue first, because it keeps the default scheduler."

Gang scheduling creates two problems of its own, and naming them is what separates a definition from experience. Fragmentation: 64 free GPUs spread as 4 per node cannot host a gang that needs 8 per node, so the gang waits while the cluster shows capacity; the fix is packing policy (fill nodes before spreading). Starvation: a stream of small jobs keeps the cluster 90% full and a 512-GPU gang never finds an instant when 512 are free; the fix is reservation (hold nodes as they free up until the gang fits) or priority aging.

fragmentation, counted
  64 GPUs free, spread as 4 free on each of 16 nodes
  a gang needing 8 GPUs per node (TP8): nodes with ≥ 8 free = 0 → cannot place
  the same 64 free GPUs packed as 8 whole nodes: place 8 such gangs
sanity: same free count, opposite outcome; the gang scheduler needs a packing policy or
        the dashboard's "64 free" is a lie for every real training job

The reversal condition: a workload of independent single-GPU tasks, where all-or-nothing admission is machinery for a constraint that does not exist and ordinary scheduling is both simpler and better packed. Gang Scheduling with Kueue and Volcano covers what the admission control actually costs. Multi-Tenancy, Quotas and Fair Share is where admission control sits, and kubectl get events on a pending gang names the reason.

What interviewers probe next

• "Kueue or Volcano?" Kueue for a Kubernetes-native cluster that also runs services, because it keeps the default scheduler; Volcano where the platform is batch-only and wants the plugins in one package.
• "One pod of a 64-gang is OOM-killed. What happens?" Without a tear-down policy the other 63 hang at the next collective; with it, the gang requeues and restarts from checkpoint, so set the policy and make the job's own timeout shorter than the scheduler's.
• "Does Slurm have this problem?" No; --nodes=32 has always been all-or-nothing, which is one reason training clusters ran Slurm for years.
• "A 512-GPU job has waited two days while small jobs keep running. What do you change?" Reservation or aging; raising its priority does nothing, because priority cannot find 512 free GPUs at one instant on a busy cluster.

Common mistakes

• Describing the deadlock as "slow" or "inefficient"; it is a live-lock with zero progress that reads as a full cluster.
• Saying "just add nodes"; the new nodes fill the same way.
• Treating gang scheduling as the whole answer and skipping the fragmentation and starvation it introduces.
• Confusing Kueue (admission in front of the scheduler) with a replacement scheduler; the distinction decides what else keeps working.

Key takeaways

• Gang scheduling: all pods placed or none, bound together, torn down together.
• Two 40-GPU jobs on 64 GPUs without it can each hold 32 forever at 0% useful work.
• Kueue admits in front of the default scheduler; Volcano and KAI replace it.
• It brings fragmentation (pack, do not spread) and large-gang starvation (reserve or age); name both.