hardNewSGLangAnthropic

Route requests to replicas by prefix using a consistent hash ring. Why virtual nodes, and how many?

A plain hash ring with one point per replica measured a 47-fold load imbalance across eight replicas. With 150 virtual nodes each it fell to 1.19-fold, and adding a ninth replica moved 11.9 percent of keys against an ideal of 11.1. Both numbers, and what they cost.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

💻 Coding for Infra

Consistent Hashing and ShardingSplitting work across N servers with a modulo of N moves almost everything when N changes, which for a cache means throwing away almost all of it. Consistent hashing places servers and keys on a ring so adding or removing one moves only its share, and virtual nodes fix the imbalance a small ring otherwise has. In LLM serving the same structure routes requests by prompt prefix so a conversation reaches the replica already holding its cache.

Advanced

📐 AI Systems Design🔒 Premium

Request Routing and Load Balancing for LLMsA load balancer for stateless web services spreads requests evenly and is done. A router for LLM replicas has two things a web balancer never had to think about: each replica holds a cache (the KV pages of recent prefixes) that makes some replicas far cheaper than others for a given request, and each request costs a wildly different amount, so counting connections is meaningless. This page builds the router that handles both: prefix-aware placement with load-aware fallback, cost-aware queue estimates, session affinity, and the failure handling when a replica restarts and its cache is gone.

Foundational

📐 AI Systems Design

Multi-Region Serving and FailoverRunning inference in more than one region buys latency for distant users and survival when a region fails, and it costs a second fleet that must be capable of absorbing the first one's traffic. The design turns on three decisions: whether regions are active-active or active-passive, what state has to cross regions and what deliberately does not, and how much headroom each region carries so a failover does not simply move the outage.

Advanced

💻 Coding for Infra🔒 Premium

Batching Queues and BackpressureWrite a request batcher is the coding round's version of the serving engine's scheduler: requests arrive one at a time, the GPU wants them in groups, and the batcher decides when a group is full enough to send without holding anyone too long or accepting more than it can hold. The two knobs are the maximum batch size and the maximum wait, the invariant is a bounded queue, and the follow-ups (priorities, cost-aware batching, cancellation, bounded in-flight batches) are the ideas the real engines carry. This page implements the batcher in asyncio, derives what each knob buys, and walks the follow-ups.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackImplement sampling from a model's logits with temperature, top-k and top-p. What are the numerical traps?Next in this trackCompute per-tenant GPU-hours for a billing period from a log of allocation start and stop events.Next in this trackImplement a circuit breaker in front of a model endpoint. What are the states, and what does the half-open state buy?

FEDITOR'S NOTE

Scored on virtual nodes as the fix for imbalance with a measured spread, on the fraction of keys that move on membership change, and on why the ring is right for cache affinity rather than for load balance alone.

DISCUSSION · 0

No comments yet — be the first to share your approach.