Splitting prefill from decode answers a problem you must have first

Putting the two phases on separate machines lets each be sized and tuned alone, and pays for it with a cache transfer between them plus a second queueing boundary. It is the right answer at scale with heavy interference, and the wrong one almost everywhere else.

15 MIN · PREMIUM