← 📐 AI Systems Design
Advanced
Request Routing and Load Balancing for LLMs
A load balancer for stateless web services spreads requests evenly and is done. A router for LLM replicas has two things a web balancer never had to think about: each replica holds a cache (the KV pages of recent prefixes) that makes some replicas far cheaper than others for a given request, and each request costs a wildly different amount, so counting connections is meaningless. This page builds the router that handles both: prefix-aware placement with load-aware fallback, cost-aware queue estimates, session affinity, and the failure handling when a replica restarts and its cache is gone.
Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew
RELATED CONCEPTS
LESSONS THAT TEACH THIS
PRACTICE THIS IN REAL QUESTIONS
LLM Inference & ServingHow do you route requests across replicas to maximize prefix-cache hits without unbalancing the fleet?→AI Infrastructure System DesignDesign an internal LLM gateway for a company: routing across providers and models, caching, fallbacks, and cost controls.→Distributed Training & ParallelismYour MoE router sends 90% of tokens to 10% of the experts. What happens to the step, and how do you fix it without hurting the model?→Coding for InfraRoute requests to replicas by prefix using a consistent hash ring. Why virtual nodes, and how many?→Distributed Training & ParallelismExplain expert parallelism for an MoE model: what the all-to-all is, how much it moves, and why load balance is the whole problem.→Kubernetes, Slurm & GPU SchedulingWe want to run LLM inference on Kubernetes. What does the deployment actually need beyond a Deployment with a GPU limit?→