TL;DR: Represent a fraction as a MIG profile (a hardware slice with memory and fault isolation, fixed shapes per GPU) for anything with an SLO or a tenant boundary, and as a soft share (memory limit plus a compute weight, enforced by MPS or time-slicing) for one team's bursty work. Place with a best-fit score that fills a GPU before opening another and fills a node before opening another, and reserve whole nodes for gangs. Count fragmentation as "requests that fit by total but not by shape," and run a defragmentation pass that migrates or restarts small jobs to consolidate. Kubernetes gets there with the device plugin's MIG resources or a DRA driver plus Kueue; KAI Scheduler ships the fractional logic ready-made.
How to approach it
Ask three things: who shares (tenants or one team), what the smallest unit of demand is (a 10 GB model, a notebook), and whether multi-GPU gangs share the same fleet. Then give the representation, the placement algorithm, and the fragmentation control, in that order, with a worked example for each. Say which parts Kubernetes already has and which you would build. Close with the failure the design still has and how you would watch for it.
A strong answer
A typical situation: a serving platform runs 200 small models on 64 H100s, each model needs 8 to 30 GB and a fraction of the compute, and the team also runs 8-GPU fine-tuning gangs on the same fleet. Whole-GPU allocation leaves most compute idle; naive sharing leaves the gangs unable to find a whole node.
Representation. A fraction has to say what it isolates. Two kinds:
| Kind | Unit | Isolation | Enforced by | Use |
|---|---|---|---|---|
| hard slice | MIG profile (1g.10gb to 7g.80gb) | memory, faults, SMs | hardware | tenants, SLOs |
| soft share | memory bytes + compute weight | memory limit only | MPS or time-slicing | one team's bursty jobs |
The scheduler tracks per GPU: its mode (whole, MIG with a profile layout, or shared), the slices or bytes free, and the node it is on. MIG, MPS and Time-Slicing has the isolation detail; the design decision is that a GPU is in exactly one mode at a time, because reconfiguring MIG drains the GPU.
fleet layout for the 200-model case
demand: 120 models at ≤ 10 GB, 60 at ≤ 20 GB, 20 at ≤ 40 GB; all latency-bound, several tenants
choose MIG (SLOs and tenants rule out soft shares)
per GPU options: 7 × 1g.10gb | 3 × 2g.20gb | 2 × 3g.40gb | 3g.40gb + 4g.40gb
small: 120 ÷ 7 = 17.1 → 18 GPUs as 1g.10gb (126 slices, 6 spare)
medium: 60 ÷ 3 = 20 GPUs as 2g.20gb (60 slices, 0 spare)
large: 20 ÷ 2 = 10 GPUs as 3g.40gb (20 slices, 0 spare)
total for serving: 48 of 64 GPUs; 16 whole GPUs = 2 nodes left for the 8-GPU gangs
sanity: 200 models on 48 GPUs is 4.2 per GPU; before, 200 whole GPUs would have been
needed and the fleet has 64, so sharing is the difference between fitting and not
Placement. The default scheduler's instinct is to spread for availability; a fractional scheduler must pack. For each request, filter GPUs whose mode matches and whose free slice or bytes fit, then score:
score(gpu) = w1 × (free after placement is 0) # finish a GPU
+ w2 × (node's other GPUs already in use) # finish a node
− w3 × (node has ≥ 8 whole GPUs free) # protect gang space
pick the highest score; ties by lowest GPU index for determinism
Whole-node gangs go through gang admission (Gang Scheduling with Kueue and Volcano) and are placed only on nodes with every GPU free, so the fractional work must never nibble at a gang-capable node while another node has room. That is what the third term does.
Fragmentation. Define it so it can be measured: total free capacity that no pending request can use because of shape. Count it under spread placement to show why packing matters:
16 GPUs in 1g.10gb mode, 112 slices; 60 slices in use
spread placement: each GPU has 3 or 4 used → every GPU has 3 or 4 free
a request for a 3g.40gb-equivalent (contiguous 4 slices under a profile change): fits on 8 GPUs only
after a drain, and a drain evicts 3 or 4 tenants each time
packed placement: 60 slices fill 8 GPUs (56) plus 4 on a ninth; 7 GPUs are empty
the same request: reconfigure one empty GPU, evict nobody
fragmentation = free slices that cannot serve the pending shape
spread: 52 free slices, 0 usable for the 3g request without eviction → 52 stranded
packed: 52 free, 49 on empty GPUs → 3 stranded
sanity: same 52 free slices; the policy alone decides whether 3 or 52 of them are useful
Even packed placement fragments over time, because jobs finish in a different order than they started. So the design includes a defragmentation pass, the way an allocator compacts: periodically compute the stranded count, and if it exceeds a threshold, migrate soft-share jobs (they can move because they are not pinned to a slice) and, for MIG tenants, schedule a restart of the fewest tenants that frees a whole GPU, done at their next deploy rather than by eviction where possible. The scheduler exposes the stranded count as a metric, and the pass is a job that runs against it.
What exists. On Kubernetes, MIG slices are exposed as nvidia.com/mig-2g.20gb by the device plugin or as devices with a profile attribute under Dynamic Resource Allocation; Kueue provides gang admission and quotas; the packing score and the defragmentation pass are a scheduler plugin or a controller you write. KAI Scheduler carries fractional-GPU and packing logic from Run:ai and is the shortest path if the fleet is mostly small tenants.
The remaining failure: a MIG layout chosen for today's demand mix is wrong for next quarter's, and changing it evicts tenants. Watch the ratio of stranded to free slices per profile, and re-plan the layout when it climbs.
The reversal condition: hardware slices rather than soft shares. MIG profiles have fixed shapes, so the packing problem stops being continuous and becomes a bin-packing over a small set of sizes, which is a different scheduler. MIG, MPS and Time-Slicing is where those shapes come from.
What interviewers probe next
- "Why not one mode, soft shares everywhere, and pack harder?" No memory or fault isolation across tenants; one runaway allocation or one Xid resets neighbors, which an SLO customer cannot accept.
- "How does the scheduler know a request's shape?" It is part of the claim: profile for MIG, bytes and weight for soft shares; a request with no shape gets the smallest profile that fits its memory.
- "The 8-GPU gangs are starving." The protection term is too weak or the fleet is over-committed to serving; reserve a fixed number of whole nodes and let serving borrow them only under a preemptible priority.
- "What is the metric that tells you the design is working?" Stranded capacity as a fraction of free capacity, per profile, alongside per-GPU SM active; low stranded and high SM active is the target.
Common mistakes
- Representing a fraction as a float (
gpu: 0.5) with no isolation story, then discovering two tenants share a fault domain. - Spreading for availability, which is right for web replicas and wrong for GPUs, where it manufactures fragmentation.
- Letting fractional work land on gang-capable nodes because they had room.
- Designing without a defragmentation step, so the stranded count only grows.
Key takeaways
- A fraction is a MIG profile (isolated, fixed shapes) or a soft share (memory limit plus weight, no isolation); one mode per GPU.
- Pack: finish a GPU, finish a node, protect whole nodes for gangs.
- Fragmentation is free capacity no pending shape can use; spread placement strands 52 of 52 free slices in the worked case, packing strands 3.
- Defragment on a metric; use MIG resources or DRA plus Kueue on Kubernetes, or KAI for a fleet of small tenants.