Explain how a Slurm job actually gets its GPUs: GRES, partitions, cgroups, and the sbatch that lands 8 GPUs on one node.

Slurm's GPU path is four pieces: GRES declares the devices, a partition groups nodes with limits, the batch script asks for a shape, and cgroups confine the job to its GPUs. The script that gets eight GPUs on one node, the flags that scatter them across eight nodes, and how ranks find their GPU.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core

🗂️ Scheduling & OrchestrationSign in

Slurm for AI ClustersSlurm is the scheduler most large training clusters still run, because it was built for exactly this shape of work: long jobs that need many nodes at once, launched with one command, placed with knowledge of the network. A candidate for a training-infrastructure role is expected to read an sbatch script, know how GPUs are requested and enforced, and explain why a job is stuck in the queue. This page covers the model, the commands that matter, the GPU-specific configuration, and the failure modes a platform engineer meets.

Foundational

🗂️ Scheduling & Orchestration

Node Lifecycle: Drain, Upgrade and ReturnA node moves through a fixed cycle between provisioning and decommissioning, and most fleet operations are one lap around it: cordon so nothing new lands, drain so running work finishes or moves, act, validate, then return to the pool. The wall-clock cost of a fleet-wide change is dominated by draining rather than by the change itself, which makes the plan a scheduling document rather than a technical one.

Foundational

🗂️ Scheduling & Orchestration

Slurm vs KubernetesEvery GPU platform team has this argument, and the two schedulers were built for different jobs: Slurm for long, large, all-or-nothing training on bare metal; Kubernetes for many services that scale up and down. Training fleets run Slurm because gang scheduling, topology and MPI-style launch are native there; serving fleets run Kubernetes because autoscaling and rolling deploys are native there. A platform that does both picks a hybrid: Slurm on Kubernetes (Slinky, Soperator) or a batch scheduler on Kubernetes (Kueue, Volcano, KAI). The interview question is which, for which workload, and why.

Foundational

🗂️ Scheduling & Orchestration

Kubernetes GPU SchedulingKubernetes knows nothing about GPUs until something tells it. The NVIDIA device plugin advertises each node's GPUs as a countable resource, the scheduler matches a pod's request to a node with enough of them, and the container runtime wires the device in. That model is enough for one job per GPU and breaks the moment you need sharing, topology or multi-node placement, which is where Dynamic Resource Allocation, the GPU Operator and the batch schedulers come in. Knowing which layer does what is the platform interview's opening question.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWe want to run LLM inference on Kubernetes. What does the deployment actually need beyond a Deployment with a GPU limit?Next in this trackDesign the scheduling and isolation for a multi-tenant fine-tuning service: hundreds of customers, a few base models, shared GPUs.Next in this trackThe cluster dashboard says 90% allocated and 30% utilized. What is happening, how do you prove it, and what policy fixes it?

FEDITOR'S NOTE

Scored on the correct sbatch for a whole node, on knowing that --gpus alone can scatter GPUs across nodes, on cgroup confinement as the isolation mechanism, and on the srun-to-rank-to-GPU mapping.

DISCUSSION · 0

No comments yet — be the first to share your approach.