November 13, 2026 · 5 min read
When does self-hosting an open model beat a hosted API on cost?
Per-token pricing is simple until volume grows. How to compare an hourly GPU with a per-token bill, and where the break-even sits.
Hosted AI APIs charge per token, which is perfect when usage is small or unpredictable. A GPU you run yourself costs the same per hour whether it serves one request or thousands. The question is where those two lines cross.
Turn the hourly price into a per-token price
Cost per million tokens = hourly GPU price ÷ (tokens per second × 3,600) × 1,000,000. A single NVIDIA L4 instance at about $0.80 an hour, running an 8B model at a sustained 1,000 output tokens per second across concurrent requests, works out to roughly $0.22 per million tokens.
The catch is the word sustained. At 10% utilization the same GPU costs ten times as much per token. Self-hosting wins when traffic is steady enough to keep the card busy, or when you can stop instances outside working hours.
What the per-token price leaves out
- Data handling: if prompts contain customer records, contracts or source code, keeping them inside your own account can matter more than price.
- Rate limits: your own GPU has no shared quota or noisy neighbours.
- Model choice: you can run fine-tuned or specialised open models that no API offers.
- Engineering time: drivers, inference servers, scaling and monitoring are real work. This is the part a control plane should take away.
Measure, don't guess
Track time to first token, tokens per second and spend per million tokens for each deployment from the first day. After a week of real traffic the break-even is no longer an estimate.
Want to see this with your own model?
Book a demo