Inference Engineering

The mechanisms that decide throughput, time to first token and cost per token in a model serving fleet: how the scheduler and the cache allocator actually work, what to measure, and which lever to pull when a number moves the wrong way.

5 modules16 lessons~8.5 hours2 free to read

Course 1 sized a service. This one runs it, and runs it well enough that the cost per token is a number you chose rather than one you discovered.

Almost every technique here is a lever on one tension: prefill and decode want opposite things from the same device. Paging, prefix reuse, chunking, speculation and phase separation are all responses to it, and the reason to learn them as a family rather than as a feature list is that the family predicts the trade each one makes.

Five modules. The engine and the cache come first because they are the mechanisms everything else acts on. The scheduler module is where latency targets stop being aspirations and become admission decisions. Precision is treated as three separate choices rather than one, because that is what it is. The last module handles the shapes real models arrive in and the fleet decisions that follow, and the capstone produces the SLO review and runbook you would actually operate from.

The engines themselves are named only in dated blocks. What is taught here is the mechanism, which outlives whichever engine currently implements it best.

When you finish, you can

Explain a serving engine as a scheduler plus a memory allocator, and predict its behaviour from that model
Design a benchmark whose numbers survive contact with production traffic
Choose components by the layer they occupy rather than by a throughput claim
Reason about paged cache allocation, prefix reuse and eviction as capacity decisions
Trade time to first token against per-token latency deliberately, with admission control as the lever
Choose weight, activation and cache precision separately, and prove the result is still the model
Decide when splitting prefill from decode is worth its transfer cost, and when it is not
Produce an SLO review and a runbook for a fleet, in the capstone

BEFORE YOU START · Foundations of AI Infrastructure, or the equivalent: you can derive the KV cache formula, place decode on the memory-bound side of the roofline without prompting, and size a fleet from a workload description.

Start: A serving engine is a scheduler with a memory allocator attached →

Syllabus

The engine

What a serving engine is underneath the configuration: a scheduler deciding what runs each iteration and an allocator deciding what fits, plus how to benchmark it honestly and how to choose between engines by architecture rather than by headline.

The cache

The allocator half of the engine: why contiguous allocation wastes most of what you paid for, how paging fixes it, why shared prefixes are capacity you already own, and what eviction really costs when the cache is full.

The scheduler

The half of the engine that decides who advances: what an iteration-level batch actually is once it stops being a cohort, how chunking keeps one long prompt from stalling everyone, and why admission control is the only lever that protects a percentile.

The precision

Quantisation treated as what it is: three independent decisions about weights, activations and the cache, each with its own kernel requirement and its own risk, and a validation step that has to be done before any of it reaches production.

The fleet

The decisions that span more than one device: what the model's own attention design already decided for you, when to split a model across devices, when to split the phases across machines, and a capstone that produces the SLO review and runbook.