AI Infra Interviews logo
← Open-Weights Models & Serving Engines / 20
mediumNewBasetenTogether AIModal

Self-host a 753B open-weights model or call a hosted API? Work the crossover.

The self-hosted price falls with volume and the API price does not, so the two cross at a token rate. Where that crossing sits, the fixed cost floor that makes low volume expensive, and the three reasons that override the arithmetic in both directions.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

The self-hosted price falls with volume and the API price does not, so the two cross at a token rate. Where that crossing sits, the fixed cost floor that makes low volume expensive, and the three reasons that override the arithmetic in both directions.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational
🧮 Open Weights & Serving Engines
Capacity Planning for Open-Weights FleetsPlanning a fleet for a sparse open-weights model works differently from planning one for a dense model, because memory follows total parameters and throughput follows active parameters, and those now differ by more than twenty times. The sizing goes in one direction only: from a traffic forecast to tokens per second, to replicas at a measured operating point, to GPUs, to racks and kilowatts. Doing it in the other direction, from an available GPU count, produces a fleet that fits the hardware rather than the demand.
Foundational
📐 AI Systems Design
Control Plane and API Design for GPU PlatformsEvery GPU platform has a control plane, and its API is what the rest of the organization experiences as the platform. Three semantics decide whether it survives contact with a network: idempotent creation so a retried request does not launch a second job on sixty-four GPUs, cancellation modelled as intent because only the node agent can stop a running process, and cursor pagination that does not skip rows when work is created during a listing.
Foundational
🧭 Ownership & Judgment
Talking About Cost and Capacity with LeadershipInfrastructure engineers are asked to justify large numbers to people who do not share their vocabulary, and the conversations go wrong in predictable ways: a technical objection with no alternative, a forecast with no assumptions, or a cost quoted in a unit the listener cannot act on. What works is a small number of costed options, a stated recommendation, the decision needed by a date, and every figure expressed in whatever the listener actually controls.
Foundational
🧮 Open Weights & Serving Engines
Reading config.json to Size a Model You Have Never RunEvery Hugging Face model ships a config.json, and it contains enough to compute the weight footprint, the KV cache per token, the parallel degrees that divide cleanly and the minimum GPU count, before downloading a byte. Doing that derivation is a standard whiteboard exercise in serving interviews because it is exactly what an engineer does on the morning a new model lands, and the fields that matter are the same across every recent architecture.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackOne replica in a serving fleet returns tokens slower than its siblings. Find out why.Next in this trackThe model advertises a million tokens of context. What actually breaks when you serve it?Next in this trackThe model authors publish a new revision. How do you roll it out without a quality regression?
FEDITOR'S NOTE

Scored on the crossover being a volume rather than a preference, on the deployment floor for a large sparse model, and on the non-price factors in both directions.

DISCUSSION · 0

No comments yet — be the first to share your approach.