AWSBee
← All articles

November 13, 2026 · 5 min read

When does self-hosting an open model beat a hosted API on cost?

Per-token pricing is simple until volume grows. How to compare an hourly GPU with a per-token bill, and where the break-even sits.

Hosted AI APIs charge per token, which is perfect when usage is small or unpredictable. A GPU you run yourself costs the same per hour whether it serves one request or thousands. The question is where those two lines cross.

Turn the hourly price into a per-token price

Cost per million tokens = hourly GPU price ÷ (tokens per second × 3,600) × 1,000,000. A single NVIDIA L4 instance at about $0.80 an hour, running an 8B model at a sustained 1,000 output tokens per second across concurrent requests, works out to roughly $0.22 per million tokens.

The catch is the word sustained. At 10% utilization the same GPU costs ten times as much per token. Self-hosting wins when traffic is steady enough to keep the card busy, or when you can stop instances outside working hours.

What the per-token price leaves out

  • Data handling: if prompts contain customer records, contracts or source code, keeping them inside your own account can matter more than price.
  • Rate limits: your own GPU has no shared quota or noisy neighbours.
  • Model choice: you can run fine-tuned or specialised open models that no API offers.
  • Engineering time: drivers, inference servers, scaling and monitoring are real work. This is the part a control plane should take away.

Measure, don't guess

Track time to first token, tokens per second and spend per million tokens for each deployment from the first day. After a week of real traffic the break-even is no longer an estimate.

Want to see this with your own model?

Book a demo