Three hundred fine-tunes as three hundred deployments would need three hundred GPUs. Batched adapters put them on one base, and the arithmetic says what each extra adapter costs in memory and in step time.
more free answers with an account · no card
← LLM Inference & Serving / 16
hardNewAnyscaleFireworksDatabricks
How would you serve hundreds of LoRA adapters on one base model, and what does it cost in throughput?
Three hundred fine-tunes as three hundred deployments would need three hundred GPUs. Batched adapters put them on one base, and the arithmetic says what each extra adapter costs in memory and in step time.
Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.
The concepts behind this question
Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.
Advanced
Multi-LoRA ServingA LoRA adapter is a few hundred megabytes of low-rank matrices that turn a base model into a fine-tuned variant, and multi-LoRA serving runs hundreds of them on one copy of the base weights by keeping the adapters in memory and applying the right one per request inside the batch. It is how a platform serves a thousand customers' fine-tunes without a thousand deployments. The costs are an extra small matmul per layer, adapter memory and loading, and a scheduler that has to batch across adapters without starving any of them.🚀 Inference & Serving🔒 Premium
Advanced
Multi-Tenant Fine-Tuning ServiceA fine-tuning service takes a customer's dataset and a base model and returns a model, and the design problem is that many customers want this at once, cheaply, without seeing each other's data, on GPUs that must not sit idle between jobs. LoRA changes the shape: an adapter is a few hundred megabytes rather than a copy of the base, so many jobs can share a base in memory and many adapters can be served from one replica. This page designs the service end to end: the pipeline, the LoRA arithmetic that sets memory and cost, the isolation, the scheduler that packs jobs, and the serving path.📐 AI Systems Design🔒 Premium
Core
Continuous BatchingContinuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.🚀 Inference & ServingSign in
Foundational
The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.🚀 Inference & Serving
UP NEXT ON YOUR JOURNEY
Next in this trackDesign an autoscaler for GPU inference replicas that reacts to load without thrashing.Next in this trackA new replica has to load a 70B model and serve traffic in under a minute. Where do the seconds go, and how do you get there?Next in this trackWhat changes when you serve DeepSeek-V3 compared with a dense 70B model?
FEDITOR'S NOTE
Scored on the per-adapter memory number, on how heterogeneous batches run without serializing, and on an honest throughput cost with the conditions that make it larger.
DISCUSSION · 0
No comments yet — be the first to share your approach.