Because model pricing is not linear. Cached reads, cache creation, fresh input, and output tokens are priced differently, so two workloads with similar token counts can produce very different bills. Effective cost per million tokens is the more useful metric when deciding where to intervene.
Why token counts and cost diverge in practice
Token totals are only a volume measure. Billing usually reflects how that volume is divided across cached input, fresh input, cache creation, and output generation, and each of those may be priced differently. That means two requests with the same token count can still land in very different cost bands, especially when cache hit rates or response length change.
The practical mistake is to treat “tokens” as a single cost proxy. In reality, prompt reuse, repeated context, and longer completions can shift spend even when overall throughput looks stable. For teams tuning usage, the useful question is not just how many tokens were consumed, but which token classes were consumed and at what marginal rate.
What drives the gap between usage metrics and spend
The main drivers are pricing structure and workload mix. Cached reads are usually cheaper than fresh input, cache writes may carry their own charge, and output tokens often cost more than input tokens. As a result, a workload that reuses context efficiently can look expensive in raw token volume but be relatively cheap in cash terms, while a short but output-heavy workload can cost more than expected.
Model choice also matters because pricing tiers vary across providers and models. The same application may route a request to different models for different tasks, which changes the cost profile without materially changing the token count. If you compare cost only at the aggregate level, those routing effects can hide the real source of spend.
Another common source of confusion is that cache effectiveness is behavioral, not static. If prompt structure, tool output, or conversation length changes, the cache hit pattern changes too. That is why cost analysis should separate steady-state traffic from bursty or exploratory traffic, because the latter often creates the most expensive token mix.
How to read token economics without getting misled
The right unit for comparison is effective cost per million tokens, segmented by token class where the billing model supports it. That lets you see whether spend is being driven by input volume, output verbosity, low cache reuse, or expensive model routing. It also makes it easier to compare workloads that have similar total token counts but very different economic behavior.
For teams operating multiple products or agents, the best comparison is per workflow, not per month. A single monthly total can hide one feature that is cheap at scale and another that is quietly consuming the majority of budget because it produces long outputs or repeatedly rebuilds context. That distinction matters when deciding whether to shorten prompts, constrain completions, or restructure cache use.
Where a team is trying to control spend, the first lever is usually not model replacement but workload shaping. Reduce unnecessary repetition, keep reusable context stable, and cap output where the application does not need verbose responses. Those changes improve cost efficiency without requiring a wholesale change in the underlying model stack.
What practitioners should verify before changing the model or budget
Before you optimize, confirm how the provider bills each token class and whether your telemetry separates cached input from fresh input and output. If your reporting only shows total tokens, you cannot tell whether a cost spike came from cache misses, long completions, or a more expensive model path. That distinction is essential before you decide to rewrite prompts or change providers.
Also verify whether token totals are being normalized consistently across environments. Testing, staging, and production often use different prompt shapes or different model defaults, so a cost comparison can be distorted if one environment generates more cacheable traffic than the other. The cleanest signal is a like-for-like comparison at the workflow level.
Secrets Management Guide is useful here because cost surprises are often a symptom of broader control gaps around repeated context, reuse, and runtime material that should be treated as governed inputs rather than ad hoc payloads.
Risk and Threat Considerations: Cost divergence is not just a budgeting issue. When usage telemetry is too coarse, teams can miss prompt abuse, unexpected output amplification, or repeated reruns that inflate spend without being obvious in raw token totals.
Failure mechanism: Billing anomalies arise when cache behavior, output length, and model routing are aggregated into one number, which masks the specific driver of spend and prevents targeted remediation.
Impact: Teams may under-budget, misattribute cost spikes, or apply the wrong optimization, which can delay detection of inefficient or abusive usage patterns.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP API Security Top 10 addresses the attack and risk surface, while NIST CSF 2.0, NIST SP 800-53 Rev 5 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | GV.RM-01 — Risk Management Strategy | Pricing divergence creates operational and budget risk that needs a defined risk strategy. |
| Recommendation — Define a cost-risk threshold for model usage and review spend anomalies against that threshold. | ||
| NIST SP 800-53 Rev 5 | AU-6 — Audit Record Review, Analysis, and Reporting | Token-cost divergence requires analysis of usage telemetry and billing signals to find the driver. |
| RA-5 — Vulnerability Monitoring and Scanning | Mispriced or inefficient AI usage needs continuous monitoring for abnormal cost growth patterns. | |
| Recommendation — Correlate usage logs and billing data to isolate which token class is driving spend. Monitor AI usage patterns for abnormal cost growth and investigate the underlying workload change. | ||
| CIS Controls v8 | CIS-13 — Data Protection | Cost control depends on understanding and governing the data and prompts that drive repeated token use. |
| Recommendation — Reduce repeated sensitive context and govern prompt reuse to lower unnecessary token spend. | ||
| OWASP API Security Top 10 | API4 — Unrestricted Resource Consumption | Unbounded token generation and repeated requests can inflate AI service cost like any resource exhaustion issue. |
| Recommendation — Set hard limits on request volume and output length to prevent runaway token consumption. | ||
Practitioner Guidance
What to measure: Track effective cost per million tokens separately for fresh input, cached input, cache creation, and output. That is the only view that shows whether efficiency is improving or merely shifting cost between categories.
Decision rule: If cost rises while total tokens stay flat, investigate cache hit rate and output growth before changing models. If total tokens rise but spend does not, the workload is probably becoming more cache-efficient and less output-heavy.
What good looks like: A stable workload should show predictable cost per workflow, not just predictable token totals. When the same request pattern starts costing more, you should be able to point to the exact class of token that changed.
Practitioner takeaway: Token counts are an input to cost analysis, not the answer, and the most useful control point is the token mix that determines marginal price.
Related resources from NHI Mgmt Group
- Why do AI workflows often cost more than their initial business case suggests?
- Why do AI coding agent bills often exceed the expected seat cost?
- Why do agentic AI systems often cost more than teams expect?
- Why do AI cost overruns often indicate an identity or agent operating outside its intended scope?