LLM Inference Cost: How to Calculate It
Key takeaways
- Inference cost = requests × (input tokens × input rate + output tokens × output rate), with a retry factor and cache-hit discount applied from logged usage, not estimated prompts.
- Output tokens cost more than input tokens because generation is sequential and memory-bandwidth-bound, while input processing parallelizes.
- Self-hosting a 70B model beats a ~$0.90/M API price only above roughly 50-60% sustained GPU utilization; idle hours erase the advantage below that.
- Model routing by task type is the biggest cost lever: most call volume is extraction or formatting that a small model handles at a fraction of frontier prices.
- Forecast spend as cost per successful outcome, not cost per request, and reconcile the model against the actual invoice monthly because prices and context sizes drift.
LLM inference cost is what you pay per token every time a model runs: price per million input tokens times tokens in, plus price per million output tokens times tokens out, multiplied by request volume. Training cost is a one-time capital expense amortized across that ongoing spend. This guide gives you the formula and worked examples to model your own bill.
What does LLM inference cost actually consist of per token?
FLOPs per token: the physics under the price
Forward-pass compute scales roughly with parameter count: about 2 FLOPs per parameter per token, so a 70B model does ~140B FLOPs per token before attention overhead. That is the physics every provider pays for. What you pay for is that compute plus a margin.
What the provider's per-1M-token price actually covers
A list price bundles GPU time, memory bandwidth (often the real bottleneck for large models), KV cache memory that grows with your context window, datacenter overhead, and the provider's margin. Cache reads and batch APIs price some of these inputs differently, which is why the same model can show several per-1M-token rates on one page. Compare current list prices across providers on their pricing pages, e.g. Anthropic's Claude pricing.
Input vs output token pricing asymmetry
Output tokens cost more than input tokens because generation is memory-bandwidth-bound and runs sequentially, while input processing parallelizes. When you model spend, track input and output separately, and count cached input at its own rate. VERIS's docs treat these as distinct line items so attribution matches the invoice.
How does LLM training cost compare to inference cost over a model's lifetime?
Training is a one-time capital expense; inference is the recurring bill. For a widely used model, cumulative inference spend typically exceeds training cost within months of launch, because every request pays again for tokens the training run paid for once.
Why inference dominates at scale
Training cost is fixed: compute, people, failed runs. Inference cost scales linearly with traffic and never stops. A model serving millions of daily requests burns more GPU-hours in a week than its training run took, and that spend lands on your invoice, not the lab's.
A simple lifetime cost ratio you can estimate
You rarely know a lab's training bill, so estimate the ratio instead: lifetime inference cost ÷ training cost ≈ (daily tokens × price per million tokens × 365 × years in service) ÷ training cost. If your own workload's monthly inference spend is growing while your prompt sizes stay flat, traffic is driving the curve, and that is the number to control. We break down the recurring side in our guide to production AI cost control.
How do you calculate inference cost for your own workload step by step?
The formula and its inputs
Cost = requests × (input tokens × input rate + output tokens × output rate). Four numbers, all measurable from your logs: monthly requests, input tokens per request, output tokens per request, and each provider's per-1M-token rates. Pull token counts from logged usage, not the prompt you wrote — system prompts, retrieved chunks, and tool schemas all bill as input. Multiply requests by your retry factor, since repeated calls after failed tool runs are billed.
Discount input tokens by your cache-hit rate, because providers price cached input at a fraction of the base rate. Frontier-scale training runs reportedly cost tens of millions of dollars; you pay that zero times. Estimates in, estimates out; logged numbers in, a bill you can act on.
Worked example 1: RAG pipeline with growing context
20,000 requests/month, 6,000 input tokens (system prompt plus four retrieved chunks), 500 output tokens, at $3/$15 per 1M tokens: 20,000 × ($0.018 + $0.0075) = $510/month. Two more retrieved chunks add $120.
Worked example 2: agent loop with retry amplification
2,000 tasks × 8 model calls = 16,000 requests at 4,000 input/300 output tokens: 16,000 × ($0.012 + $0.0045) = $264. A 15% retry rate pushes it to about $304; the retry loop, not the prompt, is where agent bills grow. VERIS bills on this same per-request model (terms).
How have LLM inference prices changed over time and why?
Per-1M-token prices have fallen sharply across model generations. Flagship models that cost tens of dollars per million output tokens a couple of years ago now have successors priced at a fraction of that, and mid-tier models deliver similar quality for cents. The trend is consistent across providers: each generation prices lower per token than the last.
What drove the price drops
Four forces: better hardware (newer GPUs push more tokens per dollar), quantization (serving models at 8-bit or 4-bit instead of full precision), distillation (small models trained to mimic large ones), and competition (providers undercutting each other on price for comparable benchmarks). None of these slow down; all four compound.
What this means for your forecasts
Don't anchor your cost model to today's prices. Model your inference bill at current rates, then assume the same workload gets cheaper, not more expensive, as you upgrade models. The risk is the opposite of what most teams expect: volume grows faster than unit prices fall, so total spend still climbs. Track cost per successful outcome, not cost per million tokens, and re-baseline your budget (pricing tiers) every quarter against what newer models actually charge.
How do self-hosted and API-based inference costs compare?
Self-hosted cost per token: the full math
Take one A100 80GB on AWS on-demand at roughly $4.10/GPU-hour (AWS EC2 on-demand pricing). A 70B model with batching sustains around 2,000 output tokens/second on that card. At 100% utilization that is about $0.55 per 1M tokens, before you count the ML engineer hours to run vLLM, handle failover, and keep drivers current.
Scenario Utilization Cost per 1M tokens (illustrative) Fully saturated batch serving 100% ~$0.55 Realistic steady traffic 50% ~$1.10 Periodic jobs, mostly idle 15% ~$3.70 API list price, 70B-class model n/a ~$0.90
The utilization break-even point
Against a ~$0.90/M API price, self-hosting wins only above roughly 50–60% sustained utilization. Below that, amortized idle hours eat the arithmetic advantage.
The idle-GPU problem for periodic workloads
If your jobs run a few hours a day, you are paying for VMs sitting between runs, a problem builders running their own GPU clusters report constantly. The honest fix is measuring your real utilization first; we log how we instrument that in our changelog.
What levers reduce inference cost most, ranked?
Model right-sizing and routing
Most teams run a frontier model on every call, including classification and extraction steps that a small model handles fine, which makes routing by task type the biggest lever available without touching product code. Portkey, Braintrust, Helicone, and LangSmith all expose routing or fallback rules, and OpenRouter makes provider switching cheap to try. The tradeoff is quality variance, so measure cost per successful outcome, not cost per million tokens. Start by pulling a week of logs, bucketing requests by task type, and checking what fraction of calls actually need frontier-level reasoning, because teams that do this often find most volume is extraction or formatting a small model handles at a fraction of the price. Route those calls first, keep the frontier model for the genuinely hard tail, and re-measure weekly against your cost-per-outcome dashboard.
Context trimming and caching
Context window growth is the quiet bill multiplier: a RAG prompt stuffing eight chunks instead of three pays for five chunks of noise on every request. Trim retrieval to what the model actually uses, and deduplicate system prompts. Prompt caching, where the provider supports it, cuts repeated prefix costs substantially on agentic workloads with long stable system prompts.
Batching and quantization
Batch API pricing typically runs well below real-time rates; if your workload tolerates hours of latency, this is nearly free savings. Quantization matters mostly if you self-host. The honest caveat: each lever trades latency or quality, and the ranking shifts with your traffic mix. VERIS attributes spend per request so you can verify which lever actually moved the bill.
How should you forecast inference cost as traffic scales?
Forecast from cost per successful outcome, not cost per request
A per-request forecast breaks the moment retries or cache misses enter the picture. Model spend as: monthly successful outcomes × (tokens per outcome ÷ 1M) × blended price per million tokens, then multiply by a retry factor you measure, not guess. Inference prices have also fallen steeply over time, so rerun the model quarterly against current list prices; Epoch AI tracks the trend in its LLM inference price dataset. Two growth vectors matter more than traffic: context windows creeping up as conversations lengthen, and retry loops that silently multiply tokens per outcome.
Attribution and budget hard stops
Tag every call with a team and feature key, so cost per successful outcome is attributable, not a blended mystery. Set a monthly hard stop per feature that degrades gracefully (smaller model, shorter context) rather than failing. Where does VERIS fit? It handles drift detection and attribution across providers, but it does not set your hard stops for you; you define the thresholds, it surfaces what changed and why.
How do you keep the cost model honest in production?
Reconciling modeled cost against the invoice
A cost model built once is wrong within a quarter. Providers change per-1M-token pricing, your context windows grow as retrieval gets richer, and retry loops quietly multiply requests. Reconcile the model against the actual invoice monthly: attribute spend per feature or pipeline, then alert when actuals diverge from modeled cost by a threshold you set. Attribution is what turns the bill from a shock into a budget hard stop you can enforce per workload.
Where VERIS fits, and where it does not
VERIS (our product) handles this reconciliation loop: it measures production traffic across providers, attributes cost per request and per successful outcome, and flags drift between modeled and actual spend. It does not negotiate your rates, fix an inefficient prompt, or replace the formula from earlier sections. If your bill is a few hundred dollars a month, a spreadsheet and invoice review may be enough; monitoring layers earn their keep when multi-provider traffic makes manual attribution impractical.
FAQ
How much does it cost to run an LLM per 1,000 requests?
Multiply 1,000 by (input tokens per request × input rate + output tokens per request × output rate). For a RAG workload with 6,000 input and 500 output tokens at $3/$15 per 1M, that is 1,000 × ($0.018 + $0.0075) = $25.50. Add your measured retry factor before budgeting, since failed tool runs bill again.
Is it cheaper to self-host an open-weight model or pay API prices?
It depends almost entirely on utilization. Self-hosting a 70B-class model on an A100 works out near $0.55 per 1M tokens at full saturation, but $3.70 when mostly idle, versus roughly $0.90 for the API. Sustained utilization above roughly 50-60% favors self-hosting; below that, idle GPU hours eat the advantage.
Why do output tokens cost more than input tokens?
Generation runs sequentially and is memory-bandwidth-bound: each output token waits for the previous one. Input processing parallelizes across the whole prompt, so it uses GPU time more efficiently. Providers price that difference directly, which is why output rates often run several times higher than input rates on the same model.
Do cached prompt tokens actually show up as savings on the invoice?
Yes, when your workload has a long, stable prefix like a system prompt reused across calls. Providers price cached input at a fraction of the base rate, so the discount appears as a separate line item. It shows up clearly on agentic workloads with repeated prefixes; it does little for one-off requests with unique prompts.
How do I split an inference bill across teams or features?
Tag every call with a team and feature key at request time, then attribute cost per request using logged token counts and the applicable rates. Aggregate per feature monthly and compare against your modeled cost. Without per-request attribution, you get one blended number that hides which pipeline or retry loop is driving spend.