Advanced

Containers, Images and GPU Cold Starts

A GPU container is a 10 to 20 GB image whose CUDA libraries must match a host driver it did not ship with, that loads tens to hundreds of gigabytes of weights before it does anything, and that then spends a minute compiling and warming before the first request is fast. Every one of those steps is a cold-start cost, and the difference between a naive deployment (minutes) and a tuned one (seconds) is a chain of specific fixes: lazy image loading, driver compatibility done right, local weight caches, and snapshots of an initialized process. This page walks the chain with numbers.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Inference Autoscaling and Cold Starts Serverless GPU Platforms Kubernetes GPU Scheduling Parallel Filesystems vs Object Storage GPUDirect RDMA and GPUDirect Storage Serving Engines: vLLM, SGLang and TensorRT-LLM

LESSONS THAT TEACH THIS

Foundations of AI InfrastructureA GPU pod fails differently from every other pod you have run14 min→

PRACTICE THIS IN REAL QUESTIONS

Kubernetes, Slurm & GPU SchedulingOur GPU pods take four minutes to start on a fresh node and sometimes fail with 'no CUDA-capable device'. Walk me through both.→Kubernetes, Slurm & GPU SchedulingDesign a serverless GPU platform where a function that loads a 7B model cold-starts in under a second. Where does every second go today?→LLM Inference & ServingA new replica has to load a 70B model and serve traffic in under a minute. Where do the seconds go, and how do you get there?→AI Infrastructure System DesignDesign a serverless GPU platform: a customer deploys a function with a model and pays per second. Where does the cold start budget go?→LLM Inference & ServingDesign an autoscaler for GPU inference replicas that reacts to load without thrashing.→Open-Weights Models & Serving EnginesDesign a platform that serves twenty open-weights models of varying size to internal teams.→