A serving API is a contract about tokens, not about requests

Model APIs look like ordinary web APIs and behave like nothing of the sort: work is unbounded until it stops, cost is per token, cancellation must reach the scheduler, and a timeout that fires without freeing a slot leaks capacity. The contract that makes those safe.

14 MIN · PREMIUM