AI spend should be judged by efficiency and output, not invoices

At a glance

What this is: This analysis says AI spend should be managed through separate efficiency and effectiveness checks, with tokens-per-dollar, model fit, and user variance as the main signals.

Why it matters: That matters because IAM, NHI, and agent governance teams increasingly need to prove that AI-controlled access, model usage, and runtime activity are producing measurable value rather than just consuming budget.

By the numbers:

• Across about thirty engineers running agents through Agent Router, the tokens-per-dollar range across top API keys was about 20x.
• Four keys accounted for around 50% of last week's total spend, and ten keys accounted for around 80%.
• The alert-triage agent moved to a model roughly 15x cheaper with no measurable drop in output quality.

Context

AI cost governance becomes difficult when managers measure consumption but not value. A model that is expensive, overused, or badly matched to the workload can look productive in dashboards while delivering little business output.

The article also exposes a governance issue that identity and access teams will recognise: runtime systems often inherit defaults, then keep them long after the original use case changes. In agent environments, that creates a policy drift problem around model choice, key usage, and governed AI spend.

Key questions

Q: How should teams measure ROI on AI spending without overfitting the numbers?

A: Separate the question into efficiency and effectiveness. Efficiency asks whether each dollar buys enough model capability, while effectiveness asks whether the output is actually useful. Measure efficiency with tokens-per-dollar and model-fit reviews. Measure effectiveness with proxy outputs such as cost per merged PR or cost per closed ticket, then use variance across similar work to spot anomalies.

Q: When should organisations switch a workload to a cheaper model?

A: Switch when the cheaper model still meets the workload’s explicit quality bar. The right test is not whether the output feels acceptable in general, but whether it satisfies the task’s minimum standard on the same inputs. If it does, the expensive model was likely a legacy default rather than a real requirement. If it does not, the higher spend is justified.

Q: What are the signs that AI spend is becoming wasteful?

A: Watch for extreme tokens-per-dollar differences between similar workloads, heavy concentration of spend in a few keys, and large variance between users doing comparable work. Those signals usually mean a model choice has not been revisited, a workload is retrying excessively, or the team is using AI because adoption is being measured rather than because output is improving.

Q: Should teams compare AI cost against usage or against output?

A: Compare against output first, then use usage as context. Pure usage can be a positive signal early on, but later it can hide waste, looping behaviour, or poor model fit. A useful governance pattern is to measure spend by workload, compare similar users, and ask whether the output justifies the cost instead of assuming activity equals value.

Technical breakdown

Why tokens-per-dollar is the first useful signal

Tokens-per-dollar is a rough but practical proxy for how efficiently a workload converts budget into model activity. It helps separate three cases that often get blended together: a workload that genuinely needs a costly model, a workload that caches well and should be cheaper, and a workload that was never re-tuned after an early prototype phase. The metric is not a full ROI measure, but it quickly surfaces where cost is structurally too high. That is why the first pass should be workload-specific, not organisation-wide. Practical implication: rank the top spenders and test whether their model choice still matches the work. See the Identity and NHI Security Business Case Guide.

Practical implication: Rank the top spenders and test whether their model choice still matches the work.

Why output quality needs a separate bar from spend

Effectiveness is not the same as efficiency. A cheap model can be unusable if it misses the task’s quality threshold, while an expensive model can be wasteful if the workload does not need its reasoning depth. The article’s point is that the quality bar must be explicit, such as severity rating accuracy for alert triage or acceptable completion for support tasks. Without that bar, teams end up optimizing for visible activity rather than useful output. Practical implication: define the minimum acceptable output for each high-spend workload before comparing models.

Practical implication: Define the minimum acceptable output for each high-spend workload before comparing models.

Why variance is more useful than absolute ROI targets

Absolute ROI targets are often too noisy for AI-assisted work because the work itself varies. Variance, by contrast, can show when two people or two workloads are materially different despite appearing similar. That makes variance a better diagnostic for overuse, stuck workflows, or hidden complexity than a single cost threshold. This is especially relevant where managers need to distinguish productive experimentation from uncontrolled spend growth. Practical implication: compare similar users or workloads against each other first, then investigate outliers instead of chasing a universal cost target.

Practical implication: Compare similar users or workloads against each other first, then investigate outliers instead of chasing a universal cost target.

NHI Mgmt Group analysis

AI spend governance is now an access-control problem as much as a budgeting problem: once model selection, key issuance, and runtime use drift apart, teams lose the ability to tell whether a workload is justified or merely active. That is why the article’s best insight is structural, not financial: spend attribution has to be tied to a governed boundary, not scattered across keys and dashboards. Practitioner conclusion: treat AI spend review as a control over usage and entitlement, not just a finance report.

Tokens-per-dollar is the right named concept for early AI governance maturity: it gives teams a practical way to separate model fit from raw activity. The point is not that cheaper is always better, but that many workloads inherit expensive defaults long after the first deployment decision was made. Practitioner conclusion: audit the default model path before you assume the workload genuinely needs premium inference.

Variance beats averages because AI work is not uniform: two engineers can run similar tasks with very different cost profiles, and that spread often reveals the real issue. Either one workflow is stuck in retry loops or one model choice is carrying unnecessary overhead. Practitioner conclusion: investigate large spend variance as a governance signal, not a performance verdict.

Runtime governance matters because AI systems change faster than cost dashboards do: the article shows that the most expensive choice is often the one nobody revisits after prototype success. That creates governance debt, where a temporary technical decision becomes an operating assumption. Practitioner conclusion: build recurring model review into the same cadence you already use for other access and cost controls.

Managed AI boundaries should include identity-aware attribution: when AI agents operate through shared gateways and issued keys, the governance question becomes who or what consumed the budget, under which policy, and for which workload. That is where provider key security and runtime controls intersect with AI spend governance. Practitioner conclusion: align cost review with key issuance, workload ownership, and policy enforcement.

Read next: Identity and NHI Security Business Case Guide

What this signals

AI cost governance is becoming a runtime policy issue: the main risk is not just overspend, but hidden model drift that makes a once-rational decision persist long after the workload changes. Practitioners should expect AI controls to move closer to issuance, attribution, and policy enforcement rather than remain stuck in finance reporting.

Identity-aware attribution is the next useful control layer: when shared gateways and issued keys route many agents through the same boundary, the governance question becomes which workload consumed what under which policy. That is why AI spend management will increasingly intersect with access governance, workload ownership, and service-account style controls.

Variance-based review is the clearest operating signal: AI teams should stop asking for one universal ROI threshold and start asking which workloads are outliers. A workflow that burns materially more budget than its peers usually deserves investigation before it becomes accepted as normal.

For practitioners

• Audit the top spenders first List the three to five workloads or keys that account for most of the AI budget, then review model choice, caching behaviour, and workload ownership before looking anywhere else.
• Define a quality bar for each workload Write down the minimum acceptable output for each high-spend use case, such as alert severity accuracy or ticket completion quality, so model tests are judged against a real standard.
• Compare cheaper models against the same inputs Run the workload through a lower-cost model using the same prompts or tasks, score the output against the quality bar, and switch only if the cheaper option still passes.
• Use variance as the weekly diagnostic Review the spread between similar users or similar workloads, because large cost differences often reveal stuck agents, bad defaults, or unnecessary reasoning overhead.
• Tie spend review to key issuance and ownership Track which team owns each key or agent, because shared gateways without clear ownership make it harder to explain whether spend reflects legitimate work or drift.

Key takeaways

• AI spend becomes manageable only when teams separate model efficiency from output effectiveness instead of treating both as one budget problem.
• The clearest signals are workload concentration, tokens-per-dollar, and variance between similar users or similar tasks.
• The practical fix is recurring review of model fit, quality thresholds, and ownership for the highest-spend workloads.

Related NHI Mgmt Group resources: Identity and NHI Security Business Case Guide · LLM Provider API Key Security and LLMjacking Guide · SIEM Buyer's Guide · MDR Buyer's Guide

Key terms

• Tokens-per-dollar: A workload efficiency measure that shows how much model activity a budget unit buys. It is useful because it exposes when a team is paying for excess capability, poor caching, or an outdated model choice rather than for the work actually being done.
• Output effectiveness: The degree to which AI-generated work produces the intended business or operational result. In practice, this means judging the model against a task-specific quality bar, not against generic satisfaction or raw usage volume.
• Variance signal: A difference in spend or output between similar workloads or users that deserves investigation. Variance matters because it often reveals hidden model mismatch, retry loops, or workflow drift better than average metrics do.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity control to the broader governance decisions that shape access, cost, and accountability.