The open-weights model you are deploying is multimodal. What changes about serving it?

An image becomes a large number of tokens before the language model sees anything, so a request that looks small carries a prefill the size of a long document. The token arithmetic, the encoder that sits outside the usual parallelism, and the two capacity numbers that move.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🚀 Inference & Serving

Prefill vs DecodeAn LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.

Core

🚀 Inference & ServingSign in

Continuous BatchingContinuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.

Foundational

🧮 Open Weights & Serving Engines

Capacity Planning for Open-Weights FleetsPlanning a fleet for a sparse open-weights model works differently from planning one for a dense model, because memory follows total parameters and throughput follows active parameters, and those now differ by more than twenty times. The sizing goes in one direction only: from a traffic forecast to tokens per second, to replicas at a measured operating point, to GPUs, to racks and kilowatts. Doing it in the other direction, from an available GPU count, produces a fleet that fits the hardware rather than the demand.

Advanced

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodePrefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYour agent product's tool-call success rate fell from 94 to 61 percent after a deploy. Debug it.Next in this trackDesign the evaluation you run against a serving deployment, not against a model.Next in this trackEvery response must be valid JSON against a schema. How do you serve that at scale?

FEDITOR'S NOTE

Scored on images expanding into token counts that dominate prefill, on the encoder's separate parallelism, and on the capacity effects of a bimodal request population.

DISCUSSION · 0

No comments yet — be the first to share your approach.