You serve a small, a medium and a large open-weights model. How do you route requests between them?

The cheapest correct answer beats the best answer at scale, so routing is a cost problem with a quality floor. The three routing signals that work, the cascade that trades latency for cost, and the measurement that catches quality loss no latency dashboard shows.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🧮 Open Weights & Serving Engines

Capacity Planning for Open-Weights FleetsPlanning a fleet for a sparse open-weights model works differently from planning one for a dense model, because memory follows total parameters and throughput follows active parameters, and those now differ by more than twenty times. The sizing goes in one direction only: from a traffic forecast to tokens per second, to replicas at a measured operating point, to GPUs, to racks and kilowatts. Doing it in the other direction, from an available GPU count, produces a fleet that fits the hardware rather than the demand.

Foundational

📐 AI Systems Design

Multi-Region Serving and FailoverRunning inference in more than one region buys latency for distant users and survival when a region fails, and it costs a second fleet that must be capable of absorbing the first one's traffic. The design turns on three decisions: whether regions are active-active or active-passive, what state has to cross regions and what deliberately does not, and how much headroom each region carries so a failover does not simply move the outage.

Foundational

🗂️ Scheduling & Orchestration

Slurm vs KubernetesEvery GPU platform team has this argument, and the two schedulers were built for different jobs: Slurm for long, large, all-or-nothing training on bare metal; Kubernetes for many services that scale up and down. Training fleets run Slurm because gang scheduling, topology and MPI-style launch are native there; serving fleets run Kubernetes because autoscaling and rolling deploys are native there. A platform that does both picks a hybrid: Slurm on Kubernetes (Slinky, Soperator) or a batch scheduler on Kubernetes (Kueue, Volcano, KAI). The interview question is which, for which workload, and why.

Foundational

📐 AI Systems Design

Control Plane and API Design for GPU PlatformsEvery GPU platform has a control plane, and its API is what the rest of the organization experiences as the platform. Three semantics decide whether it survives contact with a network: idempotent creation so a retried request does not launch a second job on sixty-four GPUs, cancellation modelled as intent because only the node agent can stop a running process, and cursor pagination that does not skip rows when work is created during a listing.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYou need a released bf16 model at half the footprint. Walk through quantizing it yourself.Next in this trackDesign a deployment that serves a trillion-parameter model at a million tokens of context with usable latency.Next in this trackThe open-weights model you are deploying is multimodal. What changes about serving it?

FEDITOR'S NOTE

Scored on routing by measured task difficulty rather than by guesswork, on the cascade's latency cost, and on continuous quality measurement per route.

DISCUSSION · 0

No comments yet — be the first to share your approach.