← Open-Weights Models & Serving Engines / 38

hardNewBasetenModalTogether AI

Fifty teams have fine-tuned the same base model. How do you serve all of them?

Fifty full copies is fifty deployments and is unaffordable. Fifty low-rank adapters over one shared base is one deployment, because an adapter is a tiny fraction of the weights and adapters from different tenants can share a batch. The arithmetic that makes it work and the two limits that decide how far it goes.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

Fifty full copies is fifty deployments and is unaffordable. Fifty low-rank adapters over one shared base is one deployment, because an adapter is a tiny fraction of the weights and adapters from different tenants can share a batch. The arithmetic that makes it work and the two limits that decide how far it goes.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

🚀 Inference & Serving🔒 Premium

Multi-LoRA ServingA LoRA adapter is a few hundred megabytes of low-rank matrices that turn a base model into a fine-tuned variant, and multi-LoRA serving runs hundreds of them on one copy of the base weights by keeping the adapters in memory and applying the right one per request inside the batch. It is how a platform serves a thousand customers' fine-tunes without a thousand deployments. The costs are an extra small matmul per layer, adapter memory and loading, and a scheduler that has to batch across adapters without starving any of them.

📐 AI Systems Design🔒 Premium

Multi-Tenant Fine-Tuning ServiceA fine-tuning service takes a customer's dataset and a base model and returns a model, and the design problem is that many customers want this at once, cheaply, without seeing each other's data, on GPUs that must not sit idle between jobs. LoRA changes the shape: an adapter is a few hundred megabytes rather than a copy of the base, so many jobs can share a base in memory and many adapters can be served from one replica. This page designs the service end to end: the pipeline, the LoRA arithmetic that sets memory and cost, the isolation, the scheduler that packs jobs, and the serving path.

🚀 Inference & ServingSign in

Continuous BatchingContinuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.

🚀 Inference & Serving

The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackA replica restarts and loads different weights than its siblings. How did that happen?Next in this trackYou must serve a frontier open-weights model on non-NVIDIA accelerators. Plan it.Closest to this topicHow would you serve hundreds of LoRA adapters on one base model, and what does it cost in throughput?

FEDITOR'S NOTE

Scored on the adapter footprint arithmetic, on batching requests for different adapters together, and on the limits that force a full copy instead.

DISCUSSION · 0

No comments yet — be the first to share your approach.