Design the physical layer for a multi-tenant GPU cloud. What changes versus a single-tenant cluster?

Isolation has to be physical where it matters and logical where it can be, and picking the boundary wrong is either expensive or a security problem. What partitions cleanly, what does not, and the allocation unit that decides fragmentation and margin.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

📐 AI Systems Design🔒 Premium

Multi-Tenant Fine-Tuning ServiceA fine-tuning service takes a customer's dataset and a base model and returns a model, and the design problem is that many customers want this at once, cheaply, without seeing each other's data, on GPUs that must not sit idle between jobs. LoRA changes the shape: an adapter is a few hundred megabytes rather than a copy of the base, so many jobs can share a base in memory and many adapters can be served from one replica. This page designs the service end to end: the pipeline, the LoRA arithmetic that sets memory and cost, the isolation, the scheduler that packs jobs, and the serving path.

Advanced

📐 AI Systems Design🔒 Premium

Serverless GPU PlatformsA serverless GPU platform lets a customer deploy a function or a model and pay only while it runs, so the platform has to start a GPU workload in seconds, pack many customers onto shared hardware without letting them see each other, and keep enough capacity warm that a burst does not wait for a cold start. Each is a design problem with numbers: the cold-start chain and the snapshot that shortens it, bin-packing memory-sized workloads onto fixed-size GPUs, the isolation boundary and its cost, and the economics of idle capacity against cold starts. This page designs the platform and derives the trade-offs.

Foundational

🖧 Hardware & Cluster Build-Out

Colocation, Power Contracts and Site SelectionFor most organizations the constraint on deploying GPUs is not the GPUs. It is finding a hall that can deliver 100 kilowatts or more per rack, reject that heat with liquid, and sign a contract for the power years before the hardware exists. Colocation contracts price reserved capacity rather than consumption, cooling capability is what eliminates most sites, and the lead time on new electrical supply is measured in years while GPUs arrive in months.

Core

🗂️ Scheduling & OrchestrationSign in

MIG, MPS and Time-SlicingA whole H100 is far more than a notebook, a small inference service or a CI job needs, and giving each of them a card leaves most of the fleet idle. Three mechanisms share a GPU, and they differ in what they isolate: MIG partitions the hardware into up to seven slices with their own memory and compute, MPS lets several processes share one GPU's SMs concurrently with no memory isolation, and time-slicing context-switches between processes with no isolation at all. The choice is the isolation the workload needs against the utilization the platform wants.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWalk me through cabling a rail-optimized cluster. What goes where, and what goes wrong?Next in this trackThe cluster passed every acceptance test and production throughput is 20 percent below plan. Where do you look?Next in this trackRoll out a driver and firmware upgrade across a live 2,048-GPU fleet without losing a training run.

FEDITOR'S NOTE

Scored on choosing the allocation unit and its consequences, on which layers can be logically partitioned and which cannot, and on the residue and audit requirements between tenants.

DISCUSSION · 0

No comments yet — be the first to share your approach.