Fireworks AI LLM Inference & Serving interview questions

Every token a model has seen leaves a key and a value in every layer. Multiply that out for a 70B model and you will see why memory, not compute, caps how many users a GPU can hold.

A small model guesses four tokens and the big one checks them in a single step. That converts idle bandwidth into tokens, and the arithmetic tells you exactly which batch size stops it working.

Same model size, three attention layouts, a 30x spread in cache per token. The formulas are short, and they explain why DeepSeek can hold four times the context of a dense 70B on the same cards.

Weight-only int4, fp8 weights and activations, or fp8 for the cache alone: each fixes a different bottleneck. Match the method to the resource you are short of, then run the evals that catch what perplexity misses.

Prefill and decode fight over the same GPU and each ruins the other's latency. Putting them on separate pools ends the fight, at the price of shipping every request's cache across the network. The break-even is a number you can derive.

Three hundred fine-tunes as three hundred deployments would need three hundred GPUs. Batched adapters put them on one base, and the arithmetic says what each extra adapter costs in memory and in step time.

671 billion parameters that touch 37 billion per token: the memory bill is for the big number and the bandwidth bill is for the small one, until batch and routing collide. The cache is tiny, and that changes the whole fit.

At 128k tokens one request's cache is 43 gigabytes and its prefill is measured in seconds. Capacity breaks first, then TTFT, then the scheduler. Each has a fix, and the numbers say which fix you need at which length.

All three guess tokens for the big model to check. They differ in where the guess comes from, how many candidates they verify per step, and how often the target agrees. Acceptance length per unit of draft cost is the number that decides.

Two hundred milliseconds is about four thousand prompt tokens of prefill on a full node, before any queueing. Every design choice is about not spending that budget twice.

An image is not one token; it is hundreds to thousands of them after a second model has run. That changes the prefill budget, the batch shape and where the first bottleneck appears.

One forward pass reads every weight. Whether that read is the bottleneck depends on how many tokens ride along with it, and the answer is different for the two halves of a request.

A batch of eight requests finishes when the longest one does, and seven slots compute padding until then. The fix is to make the scheduling unit one decode step, and the throughput math shows why that was worth several times the hardware.

Before vLLM, engines reserved the maximum context for every request up front and wasted most of it. The fix borrowed virtual memory from the operating system, and the numbers show how much capacity it returned.

Tokens per second flatters a system that is failing its users. The metrics that matter are two latencies at a percentile and the fraction of requests that meet both.

Weights take a fixed slice, the KV cache takes the rest, and the rest divided by context is your user count. Work it in bf16 and fp8, then find out why the answer also depends on how you split the model.

One 20k-token prompt can freeze every active stream for most of a second. The fix slices it into per-step budgets, and the budget number is a trade between two SLOs you can compute.