Effective Cost Per Million Tokens

What the measure actually captures

Effective cost per million tokens is a normalised unit-cost metric, not a raw spend figure. It combines price, token mix, and caching behaviour so teams can compare workloads that generate different proportions of input and output tokens without being misled by simple token counts.

The key idea is that two workloads with the same total token volume can have very different economics. A prompt-heavy workload, a completion-heavy workload, and a cached workload will not consume budget in the same way, so the measure helps translate usage into a comparable cost baseline.

Why normalisation matters

Token pricing alone does not tell you what a workload is really costing. A model that looks cheap on paper can become expensive if it produces long outputs, misses cache opportunities, or is used in a pattern that drives repeated reprocessing of the same context.

Normalisation makes the metric useful for portfolio decisions. It lets organisations compare teams, prompts, models, and routes on an apples-to-apples basis, especially when usage varies by context window size, prompt reuse, response length, or inference pattern.

Inputs that change the number

The most important drivers are the model’s pricing structure, the balance between input and output tokens, and how often cached content is reused. Those variables can move the effective rate materially even when the published per-token price does not change.

Workload shape matters as much as model choice. A system that sends large prompts, trims responses aggressively, or benefits from stable repeated context can show a very different effective cost than a bursty, ad hoc workload with little reuse.

• Higher output share usually raises the effective rate if output tokens are priced above input tokens.
• Better cache hit rates usually lower the effective rate because repeated context is billed differently or not at all, depending on the pricing model.
• Longer prompts, repeated retries, and inefficient chaining usually increase the effective rate even when traffic volume looks stable.

How to use it in practice

This measure works best when it is paired with a workload definition that stays stable over time. If prompt templates, routing, or cache policy change, the metric should be recalculated against the new operating pattern rather than compared directly with an older baseline.

It is also a governance metric, not just a finance metric. It helps teams decide whether a model switch, prompt optimisation effort, cache strategy, or output-length control is genuinely reducing unit cost, or merely shifting where the spend appears.

Secrets Management Guide is useful adjacent reading when the workload depends on API credentials or usage keys that need to be protected while you measure spend and reuse.

Related resources from NHI Mgmt Group

• How should teams calculate the true cost per AI query?
• Why do valid credentials and session tokens remain such an effective attack path?
• Who should own cost per outcome reporting for AI programmes?
• Why do leaked API tokens create such high cost risk in AI application platforms?