Ninety percent allocated and thirty percent used is the normal failure

TL;DR: Allocation says a device has an owner. Utilisation says work is happening on it. The gap is fragmentation plus idle holders, and on a busy cluster it is routinely tens of percent of the fleet. Recovering it is usually cheaper than buying more hardware, and it is a scheduling problem rather than a hardware one.

Where you are. First lesson of the fleet course. It opens on the measurement that most often triggers the question you were hired to answer: we bought all this and it is not being used.

Two numbers that are both true

Allocation is the fraction of devices assigned to a workload. Your scheduler reports it, and it is what capacity planning tends to track.

Utilisation is the fraction actually doing work. Different source, different meaning.

A fleet at 90% allocated and 30% utilised is not a paradox and it is not a broken metric. It is four things happening at once, and they have different fixes.

Reporting one number invites the wrong fix. "Utilisation is 30%, buy fewer GPUs" and "allocation is 90%, buy more GPUs" are both derivable from the same fleet, and both are wrong.

Why indivisibility makes fragmentation dominant

Your orchestrator was built for resources it can slice. A container asking for a fraction of a core gets it, and a node with a little of everything left can host something small.

A device is handed out whole. A node with one free device out of eight cannot host a job wanting two. That single property turns capacity into a packing problem, and packing problems waste space.

32 nodes, 8 devices each = 256 devices

  jobs currently placed: a mix of 1, 2 and 4-device jobs
  free devices: 40           15.6% of the fleet is free
  but they sit as: 14 nodes with 1 free, 9 nodes with 2 free, 2 nodes with 4 free

  a job needing 8 devices on one node: PLACEABLE NOWHERE
  a job needing 4: two places
  a job needing 2: eleven places

  free capacity 15.6%, placeable capacity for the job you actually have: 0%

The fleet has plenty of room and none of it is the right shape. Free capacity and placeable capacity are different quantities, and only the second one matters to the job at the front of the queue.

This gets worse over time without intervention. Long jobs outlive short ones, so the holes left behind are increasingly scattered, and a cluster that started tidy becomes a cluster where only small jobs fit. That is the same shape as memory fragmentation and it has the same remedies.

What actually recovers it

Pack deliberately. Placing a job on the fullest node that fits leaves whole nodes free for large jobs. The opposite instinct, spreading for thermal or reliability reasons, produces exactly the scattered-hole pattern above. Both are defensible and you have to choose.

Reserve shapes. If large jobs matter, keep some nodes clear of small ones rather than hoping the packing works out. A pool that only accepts jobs above a size is crude and effective.

Reclaim idle holders. An allocation with no work on it for some period is capacity you already own. This is a policy question and the technical part is easy; the hard part is agreeing the period and what happens to somebody's session.

Drain and defragment. Move small jobs off partially-used nodes to consolidate free devices. Expensive, since it means killing or migrating work, and worth doing when the fleet has become unable to place its most important jobs.

Share the device, where isolation allows. Partitioning mechanisms let more than one workload occupy a device with varying guarantees, which changes the granularity of the packing problem. Not a default: it is a decision about isolation.

What to report instead of one number

Four numbers, because the four causes have four owners:

• Allocated fraction, which is what the scheduler assigned
• Utilised fraction, which is what the devices report doing
• Placeable capacity for the queue, meaning how many of the waiting jobs could be placed right now, which is the number that predicts complaints
• Reserved fraction, labelled, so nobody counts deliberate headroom as waste

The third is the one almost nobody has and the one that turns "we need more GPUs" into a checkable claim. If the largest waiting job cannot be placed while 15% of the fleet is free, the problem is shape, not size, and buying hardware is the expensive way to fix a packing policy.

Do this before moving on

For a cluster you can picture, work out the placeable capacity for three job sizes: one device, four devices, and a full node. Use a plausible scatter of free devices rather than assuming they are consolidated.

Then say which of the four causes dominates your gap between allocation and utilisation. Most people find they cannot say, because they have two of the four numbers, and finding that out is the useful outcome.

Go deeper

• Kubernetes GPU Scheduling is how a device gets assigned at all, which is the mechanism under everything here.
• Gang Scheduling with Kueue and Volcano is the next lesson, and the reason large jobs are harder to place than their size suggests.
• Topology-Aware Scheduling is why placement quality matters beyond whether a job fits.
• MIG, MPS and Time-Slicing is the sharing option, and what isolation each mechanism does and does not give.
• The cluster shows 90% allocated and 30% utilised is this lesson as an interview question.
• Bin packing GPUs: fragmentation and defrag is the recovery side worked through properly.

Key takeaways

• Allocation and utilisation measure different things and the gap has four distinct causes with four different owners.
• A device is indivisible, so capacity becomes a packing problem and packing wastes space.
• Free capacity and placeable capacity are different quantities; only the second matters to the job at the front of the queue.
• Fragmentation worsens on its own, because long jobs outlive short ones and leave scattered holes.
• Report four numbers, and treat "we need more GPUs" as a claim to check against placeable capacity first.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1A fleet reports 90% allocated and 30% utilised. What is the correct first response?

   Increase the fleet size, since almost everything is allocatedReduce the fleet size, since two thirds of it is idleEnable device sharing across all workloadsSeparate the four causes: fragmentation, idle holders, inefficient jobs and deliberate headroom, because each has a different owner and a different fix

2. 2Why do free capacity and placeable capacity diverge, and which one predicts complaints?

   Answer it, then reveal

3. 3Why does fragmentation get worse over time even with no change in workload mix?

   Nodes drift out of the topology as they are restartedDevice memory leaks accumulate across allocationsLong-running jobs outlive short ones, so the gaps left by finished short jobs are increasingly scattered among long-lived holders, and the cluster drifts toward a state where only small jobs fitThe scheduler's placement cache degrades