“Should we self-host an LLM or use an API?” often gets answered with a cost comparison. The familiar advice is to use the API unless you process hundreds of millions of tokens a month or cannot send data outside your environment. Once engineering time is included, running your own model can cost several times the GPU bill.
That advice can be sound when you are replacing a frontier API with a large, general-purpose open model, often 70 billion parameters or more, on rented data-center GPUs. But it leaves out a less expensive option: running a small model for a task you can define and test.
Use an API as a starting point for open-ended work: conversation, varied questions, or reasoning across unfamiliar material. It also avoids paying for idle hardware when demand arrives in bursts. For narrow, repeated tasks such as extracting invoice fields or classifying tickets, a small model trained for the task is worth testing. It may match a larger model at lower running cost while giving you control over where data goes and when the model changes. A company can use both approaches, choosing separately for each task.
Model size changes the self-hosting calculation
A common cost comparison puts a roughly 70-billion-parameter model on one or more 80 GB GPUs and compares it with a premium API tier. Some published estimates put breakeven near 500 million tokens a month and the full cost, including people and idle capacity, at three to five times hardware rental. Those figures depend on the workload and prices assumed. Someone still has to configure the serving software, monitor it, and respond when it fails. AI/TLDR LeanLM Azumo
Model size can change that calculation substantially. NVIDIA researchers estimate a 10- to 30-fold inference advantage in latency, energy, and compute for a 7-billion-parameter model compared with one of 70 to 175 billion parameters. That is an inference estimate, not a reduction in the whole operating budget. NVIDIA Research The hardware requirement shrinks too. At 4-bit precision, a seven- or eight-billion-parameter model needs roughly four to five gigabytes for weights, depending on the format. It can fit on a 24 GB GPU, though context length and concurrent requests need additional memory. Self-hosting can therefore mean one workstation card or an existing server. TensorBundle’s document product, Parsarr, uses compact OCR models on customer hardware.
A fair comparison also includes small models sold through APIs. Their prices can be low enough to erase the savings from self-hosting. In that case, running the model yourself needs another justification, such as keeping data in a particular environment or retaining a tested model version. Recalculate the costs for the model that actually meets your requirements.
Where does self-hosting break even?
Change model size, traffic pattern, and API price to compare the monthly running costs.
Self-hosted model
Two 80 GB GPUs per server · $5.00 an hour · 40 engineering hours a month
Traffic pattern
Compared with API
Self-hosting breaks even at about 571M tokens a month (≈ 285,000 tasks)
This scenario compares a large self-hosted model with a frontier API at the illustrative prices shown.
- Self-hosted at 100M
- $6,850 a month
- 1 server, including engineering time
- Frontier API at 100M
- $1,200 a month
- $12 per million tokens, input and output blended
Illustrative figures for comparison, not a quote. Prices, GPU rates, and throughput vary by provider, hardware, model, and prompt length, and change often. Excludes initial build, fine-tuning, evaluation, and purchased hardware. Throughput assumes batched serving of a narrow task. Servers run all month; engineering time is priced at $80 an hour.
Many business tasks are narrow enough for a small model
Company AI plans often sound broad: an assistant for operations, AI for customer service. The individual jobs are more specific. An accounts-payable team needs invoices turned into supplier, number, date, line items, tax, and total in a fixed format. A support team needs tickets sorted into forty categories. A logistics company needs delivery notes matched against orders. You can define the expected output for each job and check how often the model gets it right.
Varied invoice layouts and free-text tickets can make a language model useful, although some tasks are better handled by OCR, classifiers, or conventional software. Tasks with clear rules and exact calculations often need no language model at all, as Don’t hire a genius to do a calculator’s job argues. When a model does help, it may need much less breadth than a frontier model provides.
For a defined task, test whether a small model meets the required accuracy, both before and after fine-tuning. Compare it with the API models you would actually use on the same held-out examples. That comparison should determine whether the smaller model is good enough for the job.
The task must be stable enough to define with examples, and those examples must cover its awkward cases. How many you need depends on the task’s scope and variety, as How much data do you need to fine-tune an LLM? explains. A small model may struggle outside its training scope, so the system also needs a way to detect and route those requests.
Use the large model to build the small one
A frontier model can help build the specialist. During development, it can label examples for people to correct, generate difficult variations missing from the historical data, and help identify failures in candidate models. Its judgments still need checking. In production, it can handle unusual cases that the small model cannot reliably answer, while routine traffic stays on the specialist. You may use the frontier model heavily during development, then only for a fraction of production requests.
Before using generated examples for training, check the provider’s terms and the candidate model’s license. The fallback also has to follow your data rules. If a document must stay inside your environment, send difficult cases to an authorized reviewer or a larger private model. Calling an external API a fallback does not make it an approved destination.
Running the model gives you control over updates
API providers decide when to retire their models. Anthropic retired Claude Sonnet 4 on June 15, 2026, after announcing the change on April 14; its policy gives at least 60 days’ notice for publicly released models. Anthropic A replacement may improve a chat assistant. For an extractor feeding a strict schema into an audited accounting process, it also creates work: someone has to rerun the evaluation and investigate changed outputs. With weights you run yourself, you can retain a tested version while evaluating its replacement. You still have to maintain the serving software, but you control the model migration schedule.
Corrections are useful whichever route you choose. A reviewer fixes an extracted total or reroutes a ticket, and that example can improve prompts, retrieval, evaluation, or training data. Some API providers also support hosted fine-tuning, so learning from corrections is not exclusive to self-hosting. Running your own weights gives you more control over how long you keep the resulting version and where it runs. Capture the original input, the model’s output, and the corrected answer from the first release so you can use those examples later.
Choose where each task runs
A company may have several reasons to use AI, with different constraints for each task. Start with where the data is allowed to go. Then ask whether the expected output is fixed or open-ended, how steady the volume is, and whether response time or connectivity limits where the model can run. Confidential data requires an approved processing arrangement; it does not automatically rule out every API.
Invoice extraction might suit a small private model, while a research assistant answering unpredictable questions might use an API. A phone voice bot may benefit from a small model near the call infrastructure if testing shows that it responds quickly enough. A mixed deployment needs explicit rules for which data can take which route. NVIDIA researchers make a similar case for combining specialized small models with larger models where broader capabilities are needed. NVIDIA Research Much of the implementation work lies in those routing rules and connections, as discussed in You’re not doing AI. You’re doing integration.
Where should each task run?
Choose a task and adjust its conditions to see which deployment to test first.
4 of 6 include a private model; 2 use an API alone.
Invoice extraction
How sensitive is the data?
What does the output look like?
How much volume?
Any limit on where it runs?
Small private model
This is a good candidate for a small model. Compare its accuracy and operating costs with a hosted model on the same cases.
These suggestions assume approved data handling. Test model quality, latency, and cost on your own cases before choosing a deployment.
Test on your own cases
To find out whether a small model is good enough, collect a few hundred real examples, including the blurred scan, the invoice in another currency, and the ticket that fits two categories. Write down the correct answer for each. Compare a frontier API, a cheaper hosted model, and one or two small candidates on the same set, measuring quality, response time, and cost per correct result. Keep evaluation examples separate from training data. Inspect individual failures as well as averages; Perfect on paper, wrong in practice and Nothing was built to answer your exact case explain why that matters.
Set the decision rule before seeing the results. For example, choose the small model if it comes within two percentage points of the API’s task accuracy, introduces no additional failures on the critical test cases, and costs less per correct result at expected volume. Passing that sample does not establish reliability for every future case. One to two weeks can be a useful target for an initial comparison if the data is ready; assembling and checking examples may take longer. Keep a held-out test set for future model changes, and add reviewed production cases without using the same examples for both training and evaluation.
Rent the generalist, own the specialist
Model capability and hosting are separate choices. A frontier API is convenient when the work calls for broad capability, and a small trained model is worth considering when the task is narrow and repeated. Steady demand can make self-hosting economical, but a cheap hosted model may still win once you count engineering time. Keeping your own weights gives you control over deployment and upgrades; improving them takes collected corrections, training, and another evaluation.
Start with a task that runs often and has a clear right answer. Build its test set and compare the candidates at the volume you expect. If the specialist performs well enough and its operating costs make sense, you have a concrete reason to run it yourself.
