TL;DR: A cold start on a naive stack is 30 to 90 seconds: image pull, CUDA context, weight load, engine warm-up. The design attacks each: lazy-loaded content-addressed images, a node-local weight cache on NVMe, and a checkpoint-restore snapshot of the initialized process (host and GPU memory) so the common path is restore plus a few seconds. Warm pools are sized by Little's law, scale-up arrivals per second times the residual cold start, per function class. Functions are isolated in microVMs with a GPU passed through; packing several functions on one GPU is done only for small models with MIG or explicit memory limits. Billing is per second from the allocation record, and the platform's margin is idle warm GPUs.
How to approach it
Ask what a customer deploys (a container with a model, or a model reference), the model size range, the latency the customer expects on a cold request, and the tenancy model (dedicated GPUs, or shared). Say the whole design is a cold-start budget and a warm-pool policy, then draw the control plane and the node agent. Decompose the cold start with numbers, show what a snapshot buys, size the warm pool, then take isolation and packing as the deep dives. Close with billing and failure modes.
A strong answer
A typical situation: a platform hosting 20,000 customer functions on a fleet of 2,000 H100s, most functions idle most of the time, a customer expectation that a cold request returns within 10 seconds and a warm one within the model's own latency. Serverless GPU Platforms covers the general shape; the numbers below make it concrete.
The cold-start chain. Each stage has a time, and the design is a list of which stages it removes.
naive: a 7B model in bf16 (14 GB), a 10 GB container image
image pull from a registry at 1 GB/s ...................... 10 s
container start, CUDA context, library load ............... 5 s
weights from object storage at 1 GB/s ..................... 14 s
engine warm-up (graph capture, first kernels) ............. 10 s
total ≈ 40 s; a 70B replica on 8 GPUs runs well past 60 s
after each fix
lazy image loading (fetch blocks on demand from a content-addressed store) ... 10 s → ~1 s
node-local weight cache on NVMe at 7 GB/s: 14 GB → 2 s; PCIe to HBM at ~25 GB/s → 0.6 s
snapshot restore: the initialized process, host memory and GPU memory captured after warm-up,
restored from NVMe: 14 GB weights + ~4 GB process at 7 GB/s ≈ 2.6 s, and warm-up is skipped
hot path ≈ 3 to 4 s for a 7B; a 70B on one node ≈ 141 GB ÷ 7 GB/s per NVMe ≈ 20 s unless striped
sanity: the snapshot removes the two largest stages (weights and warm-up); everything else is
under a second, so the restore bandwidth from NVMe is the number the whole platform sits on
Warm pools. A warm instance is a restored process holding a GPU with no request in flight. It costs $2.50 per hour idle and saves the residual cold start on every scale-up.
Little's law: warm instances needed = scale-up arrival rate × cold start time
a class of functions that together trigger 20 scale-ups per second at peak, residual cold start 4 s
→ 80 warm instances hold the queue at zero; at 10 s naive cold start it would be 200
cost: 80 × $2.50 = $200/h at peak, against 2,000 GPUs × $2.50 = $5,000/h of fleet
per function: keep a minimum of 0 or 1 warm depending on the customer's plan; predictive pre-warm
from the function's own daily pattern for the top 5% by traffic
scale-down: an instance idles for a grace period (say 60 s) before its GPU is released; shorter grace
raises scale-up arrivals, longer grace raises idle cost; tune per class from the measured arrival rate
Isolation. The deep dive interviewers want. Each function runs in a microVM (Firecracker-class) with the GPU passed through, because customers ship arbitrary code and a container shares the host kernel. The cost is a slower snapshot (the VM's memory rather than a process) and one GPU per VM, since passthrough gives the whole device. Packing several small functions on one GPU needs either MIG partitions (up to seven on an H100, each with fixed memory and compute) or time-slicing with memory limits, and time-slicing has no performance isolation: one tenant's long kernel stalls another's. The decision: microVM per GPU for anything over 10 GB of weights, MIG partitions for small models, no time-slicing across customers.
Bin packing. The placer packs microVMs onto nodes by GPU count and by weight-cache locality: a function whose weights are already on a node's NVMe restores in 3 s; on a cold node it pays the fetch. Score candidate nodes by (weights cached, free GPUs, same-customer affinity for multi-GPU functions) and keep whole nodes free for 8-GPU functions, the same fragmentation logic a training scheduler uses.
Billing. Per-second charge from the allocation record on the node agent, with an idempotency key per (allocation, second), reconciled against GPU telemetry; customers are charged for warm time only if they asked for a minimum warm count.
The trade-off to commit to: microVM per GPU rather than shared-GPU containers. It costs the platform packing efficiency on small models and buys the isolation that lets it take arbitrary customer code. The reversal condition: a platform serving only its own curated models (no customer code) can pack containers with memory limits on one GPU and roughly triple density for small models. Serverless GPU Platforms covers the cold-start chain, Containers, Images and GPU Cold Starts has the per-stage costs, and p99 time to first token from a cold pod is the number the product is sold on.
Failure modes to name: a snapshot taken from an unhealthy state that restores broken everywhere (validate a snapshot with a probe request before it is published); a weight-cache eviction storm when a popular model's new version lands (pin by reference count); a node's NVMe filling with stale snapshots (LRU with a floor of free space); a customer function that never returns and holds a GPU (max duration, enforced); a scale-up burst larger than the warm pool (queue with a visible estimate, then reject with retry-after).
What interviewers probe next
- "Why not keep every function warm?" 20,000 functions on 2,000 GPUs cannot all be warm; the warm pool is sized from arrivals × cold start, and the idle cost is the platform's margin.
- "What does the snapshot actually contain?" The process after warm-up: host memory, the GPU's memory and context, open file descriptors re-established on restore; anything holding a network connection is reopened by the runtime after restore.
- "How do you handle a 70B on 8 GPUs?" Stripe the restore across eight NVMe devices in parallel (8 × 7 GB/s), keep the snapshot node-local, and treat it as a class with its own longer cold start and a minimum warm count.
Common mistakes
- Quoting one cold-start number with no chain, so nothing can be optimized.
- Snapshotting before warm-up, which saves the weight load and keeps the 10 s of graph capture.
- Time-slicing GPUs across customers and calling it isolation.
- Sizing the warm pool as a fixed percentage of the fleet rather than from arrival rate × cold start.
Key takeaways
- Cold start = image + CUDA + weights + warm-up; lazy images, NVMe caches and a post-warm-up snapshot cut 40 s to about 4 s for a 7B.
- Restore bandwidth from NVMe (about 7 GB/s per device) is the number the platform sits on; stripe for large models.
- Warm instances = scale-up arrivals per second × residual cold start; 20/s × 4 s = 80.
- MicroVM per GPU for customer code; MIG for small models; never time-slice across customers.