← 🚀 Inference & Serving
Core
Continuous Batching
Continuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.
a free account unlocks the core curriculum tier · no card
RELATED CONCEPTS
LESSONS THAT TEACH THIS
Foundations of AI InfrastructureA request becomes tokens, and tokens become two different workloads12 min→Foundations of AI InfrastructurePrefill is a matrix multiply; decode is a memory read wearing a matmul's clothes14 min→Inference EngineeringA serving engine is a scheduler with a memory allocator attached13 min→
PRACTICE THIS IN REAL QUESTIONS
LLM Inference & ServingWhat is the difference between static and continuous batching, and why did it change LLM serving?→AI Infrastructure System DesignDesign an LLM batching system end to end: the queue, the batch, the KV cache and streaming. Give me numbers.→Coding for InfraImplement the scheduler loop of a continuous-batching engine: admit, step, preempt. Where does it livelock?→Open-Weights Models & Serving EnginesThe engine logs say CUDA graphs are disabled for your deployment. Does it matter?→Open-Weights Models & Serving EnginesIs speculative decoding worth enabling on a trillion-parameter mixture-of-experts model?→LLM Inference & ServingYou have one GPU and a synchronous API that receives 100 documents at once. Design the batching, and show the latency math.→