hard★ EssentialNewBasetenModalAnyscale

Design an autoscaler for GPU inference replicas that reacts to load without thrashing.

A replica takes a minute to become useful and the traffic changes in seconds. Scale on the wrong signal and you buy GPUs you never use, or you drop requests while they boot. The design is three numbers and two timers.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🚀 Inference & Serving🔒 Premium

Inference Autoscaling and Cold StartsScaling an LLM fleet is harder than scaling a web service because a replica takes minutes to become useful (pull an image, load 141 GB of weights, warm the cache) and costs several dollars an hour while idle. The signals that work are queue depth and TTFT against the SLO, not GPU utilization, which is misleading for memory-bound decode. The design is a warm pool sized for the burst, hysteresis so the fleet does not thrash, and a cold-start path measured in seconds through snapshots and weight streaming.

Advanced

📐 AI Systems Design🔒 Premium

Serverless GPU PlatformsA serverless GPU platform lets a customer deploy a function or a model and pay only while it runs, so the platform has to start a GPU workload in seconds, pack many customers onto shared hardware without letting them see each other, and keep enough capacity warm that a burst does not wait for a cold start. Each is a design problem with numbers: the cold-start chain and the snapshot that shortens it, bin-packing memory-sized workloads onto fixed-size GPUs, the isolation boundary and its cost, and the economics of idle capacity against cold starts. This page designs the platform and derives the trade-offs.

Foundational

🚀 Inference & Serving

Latency Metrics: TTFT, TPOT and GoodputAn LLM request has two latencies, not one: time to first token, set by queueing and prefill, and time per output token, set by the decode loop. Reporting them as percentiles, and reporting goodput (requests that met both SLOs per second) rather than raw throughput, is what separates a serving engineer from a benchmark reader. The numbers a loop expects: about 24 tokens per second single-stream for a 70B model on one H100, TTFT floors in the hundreds of milliseconds for long prompts, and p99s that come from queueing, not from the GPU.

Core

🩺 Fleet Reliability & ObservabilitySign in

SLOs for AI SystemsA service level objective is a promise with a number attached, and AI systems need their own because the classic ones do not fit: a training run has no requests, only progress, so its objective is goodput; an LLM endpoint streams, so its latency is two numbers (time to first token and time per token) rather than one; and both spend a budget that is set by hardware failure rates rather than by software bugs. This page defines the objectives that fleet and serving teams actually use, derives the thresholds from user needs and from the hardware, and works the error-budget arithmetic that decides when to stop shipping and start fixing.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackA new replica has to load a 70B model and serve traffic in under a minute. Where do the seconds go, and how do you get there?Next in this trackWhat changes when you serve DeepSeek-V3 compared with a dense 70B model?Next in this trackHow do you route requests across replicas to maximize prefix-cache hits without unbalancing the fleet?

FEDITOR'S NOTE

Scored on choosing signals that lead the SLO breach, on pricing the cold start against the idle cost with numbers, and on the asymmetric hysteresis that stops the oscillation.

DISCUSSION · 0

No comments yet — be the first to share your approach.