Advanced

Activation Checkpointing

Activation checkpointing discards a layer's intermediate tensors during forward and recomputes them during backward, trading up to a third more compute for an order-of-magnitude cut in activation memory. Selective recomputation keeps the tensors that are expensive to recompute and cheap to store, and the extra FLOPs show up as HFU without adding a token of progress.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

ZeRO and FSDP MFU and HFU Context and Sequence Parallelism Tensor Parallelism Model Memory Footprint GPU Memory Hierarchy

LESSONS THAT TEACH THIS

Training at ScaleMemory decides your parallel layout before you get an opinion14 min→Training at ScaleSharding is what makes a large model fit, and it is not free15 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismWhen does activation checkpointing pay for itself, and what would you checkpoint first?→Distributed Training & ParallelismYou need to train a 100B dense model and it does not fit on one node. Walk me through how you would lay it out.→Distributed Training & ParallelismExplain ZeRO stages 1, 2 and 3. How much memory does each stage leave per GPU for a 70B model?→LLM Inference & ServingWhat is the KV cache, and why does it keep growing while a request is being served?→Napkin Math, Cost & CapacityHow much GPU memory does it take to run Llama 3.1 70B?→Napkin Math, Cost & CapacityHow big is the KV cache for Llama 3.1 70B at a 128k context?→