Self-hosting an open-weight model: the break-even is further away than you think

Someone computes the arithmetic on a whiteboard. A GPU instance costs a certain amount per hour, it can produce a large number of tokens per second, therefore the cost per million tokens is a fraction of what the API charges. The conclusion looks obvious.

The flaw is the same one that makes on-premise hardware look cheap against cloud in general. The API bills for tokens you actually used. The GPU bills for time, whether the queue is full or empty. At 5 percent utilisation, which is what a typical internal tool achieves, the effective cost per token is twenty times the whiteboard number.

Utilisation is the variable that decides this, and almost nobody estimates it before deciding.

The structure of each cost

API pricing is per input and output token, with output typically costing several times input. It is strictly variable: zero traffic, zero cost. Prompt caching cuts the cost of a repeated system prompt or a stable document context substantially, and batch or asynchronous tiers cut it further for work that can wait. Those two mechanisms are frequently ignored, and they change the comparison more than most architectural decisions do.

Self-hosting is per GPU-hour, plus the engineering to run it. Fixed with respect to traffic and continuous. A reserved instance or a commitment lowers the hourly rate and increases the fixed portion, making utilisation matter even more.

So the honest comparison is not cost per token against cost per token. It is your monthly API spend against the monthly cost of enough GPU capacity to serve your peak, multiplied by the redundancy you need, plus a fraction of an engineer.

Utilisation is the whole argument

Work out the tokens per second you actually need at peak, and then at the median. The ratio between them is your problem.

A batch workload that processes a queue overnight can run a GPU at high utilisation, and it is the best case for self-hosting by a wide margin. An interactive assistant used by staff during working hours in one time zone is idle for most of the week, and it is close to the worst case unless you can scale to zero, which for a large model means cold starts measured in minutes because the weights have to be loaded.

Two consequences follow. First, mixed workloads help: if you can push batch work onto the same hardware during quiet hours, the economics change substantially. Second, peak-to-median ratio decides whether you can buy reserved capacity or have to pay on-demand rates for burst, and GPU on-demand availability is itself not guaranteed in every region.

What actually determines throughput

Naive serving with a simple inference loop wastes most of the hardware. The difference between a poor deployment and a good one is several times the throughput on the same GPU, which moves the break-even more than the choice of instance does.

Continuous batching is the largest single factor. A purpose-built inference server processes many requests concurrently, adding new ones to the batch as others finish, instead of waiting for a batch to complete. Without it you are paying for a GPU that is mostly idle between tokens.

Memory determines everything else. The weights occupy memory proportional to parameter count and precision. What people forget is the attention cache, which grows with the number of concurrent requests and the length of each context. A long-context workload with many concurrent users can spend more memory on cache than on weights, and when it does not fit, throughput collapses.

Quantisation reduces weight memory and increases throughput at some quality cost. Eight-bit is generally close to lossless for most tasks. Four-bit is usable and the degradation is task-dependent, so it has to be measured on your own evaluation set rather than assumed from a benchmark. Quantisation is often what makes a model fit on one GPU instead of two, which is a step change in cost rather than an incremental one.

Model size is the biggest lever of all. A small model that is good enough for your task, possibly distilled from a larger one, changes the economics completely. The reasoning for that is in fine-tuning, RAG or a better prompt, and it is the most underused option in this whole area.

The costs outside the instance

• Redundancy. One GPU is a single point of failure. Two is the minimum for anything production, which doubles the fixed cost before you serve a single extra request.
• Non-production. A staging environment with a GPU is real money, and sharing one with development is the usual compromise.
• Storage and loading. Model weights are large, and every scale-out event reads them. Cold start time is a product decision, not just an ops detail.
• Engineering time. Someone owns the inference server, the upgrades, the capacity planning and the incidents. Budget a fraction of a person permanently.
• Evaluation. With an API you inherit the provider's quality work. Self-hosting means you own the evaluation set and the regression testing, as in evaluating an LLM feature.
• Quota and availability. GPU capacity is constrained in many regions, and a quota request is a lead time, not a click.

The reasons that are not about cost

These are the good reasons, and they justify self-hosting even when the arithmetic does not.

Data residency and control. If the data cannot leave your boundary, or a buyer's questionnaire requires it, the model has to run where you control it. This is the most common legitimate driver we see in European engagements, and it connects directly to where your data actually sits.

Version stability. A provider can deprecate or silently update a model behind a stable endpoint. A self-hosted model is exactly the weights you deployed, which matters when you have a validated system and a regulator asking how its behaviour is controlled.

Latency and locality. Running close to the workload removes a network round trip, which matters for interactive or high-frequency use.

Customisation. Serving several fine-tuned adapters on one base model is straightforward when you own the server and often unavailable or expensive otherwise.

Predictability. A fixed monthly cost is sometimes worth more to a finance function than a lower variable one.

The shape that usually wins

For most organisations the answer is neither pure option.

Use an API for interactive, spiky, low-volume and quality-sensitive work, with prompt caching enabled and a small model for the easy subset of requests. Self-host for high-volume, steady, narrow tasks: classification, extraction, embeddings, batch enrichment. Embeddings in particular are a strong self-hosting case, because the models are small, the work is often batch, and the volume is high.

Then route by task rather than choosing one provider for everything. The routing layer is a small piece of engineering that pays for itself quickly, and it keeps the exit available in both directions.

What to do this week

Take last month's API bill and divide it by the hourly cost of the GPU instance you would need for your peak, times two for redundancy. That number is the fraction of a month you would have to run at full utilisation to break even. Then look at your traffic graph and ask honestly what your utilisation would be. In most estates that comparison settles the question in an hour. We run it in the cost phase of an applied AI project.