Your fine-tunes run on spot GPUs preempted about once every four hours. How often should they checkpoint, and when does spot stop paying?

Spot is 60% cheaper until the preemptions eat the discount. The expected-loss formula (waste per preemption = half the checkpoint interval plus restart), the break-even worked at three cadences, the checkpoint cost that pushes back the other way, and the gang arithmetic that says which jobs must never be on spot.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Spot, Preemption and Capacity StrategiesSpot and preemptible GPUs cost a fraction of on-demand and can be taken back with a couple of minutes' notice, so using them well is an expected-value calculation: the discount against the work lost per preemption, which is set by checkpoint cadence and restart time. The same arithmetic governs internal preemption in a shared cluster. This page works the break-even, the checkpoint interval that makes spot pay, and the fleet mix (reserved baseline, on-demand headroom, spot for tolerant work) that a capacity strategy is built from.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Multi-Tenancy, Quotas and Fair ShareA shared GPU pool is cheaper than ten private ones because ten teams' demand is smoother than one team's, and it only works if the sharing is enforced. Quotas say what each team is guaranteed, borrowing lets idle guarantees be used by others, fair share decides who waits when everyone wants more, and preemption reclaims borrowed capacity. This page works the arithmetic that makes pooling worth it, the layers of isolation a tenant needs, and the incentive problems (hoarding, gaming, the research-versus-product tension) that any policy has to survive.

Advanced

🚀 Inference & Serving🔒 Premium

Multi-LoRA ServingA LoRA adapter is a few hundred megabytes of low-rank matrices that turn a base model into a fine-tuned variant, and multi-LoRA serving runs hundreds of them on one copy of the base weights by keeping the adapters in memory and applying the right one per request inside the batch. It is how a platform serves a thousand customers' fine-tunes without a thousand deployments. The costs are an extra small matmul per layer, adapter memory and loading, and a scheduler that has to batch across adapters without starving any of them.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Gang Scheduling with Kueue and VolcanoA distributed training job is 64 pods that start together or not at all: if 40 are running and 24 are Pending, the 40 hold their GPUs idle at a collective barrier waiting for ranks that may never come, and two such jobs can deadlock a whole cluster. Gang scheduling makes the job the unit of admission. Kueue and Volcano add queues, quotas, priorities and preemption on top, which is what turns a pile of GPUs into a platform several teams can share without starving each other.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackOne fleet: training that wants every idle GPU, and inference with a p99 SLO. Separate pools, or one pool with preemption? Show the numbers.Next in this trackDesign a job queue for 100k GPU jobs with preemption: what state, what ordering, and what happens when a quota owner returns?Next in this trackWhat is Ray on Kubernetes good for, and where do its scheduler and the Kubernetes scheduler fight each other?

FEDITOR'S NOTE

Scored on writing the waste fraction as λ × (T/2 + R) and solving it, on including the cost of checkpointing itself so the interval is an optimum rather than 'as short as possible', and on the gang-level preemption rate.

DISCUSSION · 0

No comments yet — be the first to share your approach.