Advanced

PagedAttention

PagedAttention stores the KV cache in fixed-size blocks scattered across HBM and maps each sequence's logical positions to physical blocks through a block table, the same trick an operating system uses for virtual memory. It removes the reservation and fragmentation waste of contiguous allocation, lets blocks be shared between sequences, and is why an engine can decide admission by counting free blocks.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

The KV Cache Continuous Batching Prefix Caching and KV Reuse Chunked Prefill GPU Memory Hierarchy KV Cache Sizing

LESSONS THAT TEACH THIS

Inference EngineeringA serving engine is a scheduler with a memory allocator attached13 min→Foundations of AI InfrastructureThe KV cache is what actually decides how many users fit14 min→Foundations of AI InfrastructureBatching is a scheduling decision, not a configuration flag13 min→

PRACTICE THIS IN REAL QUESTIONS

LLM Inference & ServingHow does PagedAttention work, and what problem was it solving?→CUDA, Triton & Kernel EngineeringImplement the verification step of speculative decoding, including the rollback of the KV cache after a rejection.→CUDA, Triton & Kernel EngineeringSketch a paged attention kernel. What changes from FlashAttention once the KV cache is not contiguous?→Open-Weights Models & Serving EnginesSet max-model-len and max-num-seqs for a chat product from first principles.→Open-Weights Models & Serving EnginesvLLM crashes with out of memory during startup on a model that should fit. Debug it.→Open-Weights Models & Serving EnginesIs speculative decoding worth enabling on a trillion-parameter mixture-of-experts model?→