← 💻 Coding for Infra
Core
Rate-Limiting Algorithms
A rate limiter answers one question, 'may this request proceed now?', and the three classic algorithms answer it with different shapes of fairness and memory: the token bucket allows bursts up to a capacity and refills at a rate, the leaky bucket smooths output to a fixed rate, and sliding windows count recent requests exactly or approximately. AI platforms limit in tokens as well as requests, per tenant, across many gateways, which adds two twists: a request's cost is unknown until it finishes, and the counters must be shared. This page derives each algorithm, implements the token bucket correctly, and covers both twists.
a free account unlocks the core curriculum tier · no card
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
Coding for InfraImplement a token bucket rate limiter. Make it thread-safe, and explain what the two parameters actually control.→AI Infrastructure System DesignDesign rate limiting for an LLM API. Why tokens instead of requests, and how does a bucket work when the cost is unknown until the end?→CUDA, Triton & Kernel EngineeringGiven the addresses each thread in a warp touched, classify the access pattern: coalesced, strided or random. Write the classifier.→Coding for InfraPlace GPU jobs onto nodes. Compare first fit, best fit and worst fit, and say which one a training cluster wants.→Coding for InfraImplement a scheduler that admits jobs by priority and preempts lower-priority work when it must. What are the rules?→CUDA, Triton & Kernel EngineeringWrite a fused row softmax in Triton, explain why it is one HBM pass, and say where it stops scaling.→