AI token spend obscures whether agents are delivering business value

At a glance

What this is: Arize argues that AI token spend is a poor proxy for value because rising usage and falling per-token prices can mask whether agents actually deliver shipped work, resolved tickets, or accepted changes.

Why it matters: This matters because security and identity teams increasingly need to govern AI agents as production systems, which means measuring outcomes, not just consumption, when deciding whether access, spend, and automation are justified.

Context

AI spend is easy to count, but AI value is harder to prove. When token prices fall while usage rises, billing data can suggest efficiency even when the underlying work is not getting delivered, which leaves governance teams with a weak control signal. For programmes that now include AI agents, that gap matters because cost, access, and output all need to be measured against an actual business result.

The problem is not limited to finance. In agentic workflows, the same system that consumes tokens may also open pull requests, trigger tools, or move data, so a narrow cost metric misses both delivery quality and control risk. For identity and AI governance teams, the more relevant question is whether the agent produced a verifiable outcome that justifies the access and compute it used.

Key questions

Q: How should teams measure whether AI agents are actually delivering value?

A: Teams should measure cost per outcome, not just token consumption. That means tying AI spend to a verifiable result such as an accepted pull request, resolved ticket, or shipped feature. The metric only works when the run is traced and the result is evaluated, so observability and outcome scoring are part of the measurement system.

Q: Why does lower token pricing not prove that AI is becoming more efficient?

A: Lower token pricing only shows that a unit of model output is cheaper. It does not show whether agents are using more context, making more calls, or producing outcomes that matter. Total spend can still rise if the workflow expands faster than unit cost falls, so efficiency has to be judged on accepted work, not token volume.

Q: What breaks when AI traces are kept separate from evaluation workflows?

A: Iteration slows down because teams have to export data manually, rebuild context in other tools, and lose the feedback loop between production behavior and quality improvement. AI observability works best when traces, evaluations, datasets, and experimentation live together. Without that connection, teams can see problems but cannot quickly test fixes against real interactions.

Q: What should identity and AI governance teams do when agent access is hard to justify by output?

A: They should treat access as conditional on measurable results. If an agent cannot demonstrate accepted work, its permissions, budget, and scope should be reviewed together, because unchecked access with no outcome evidence creates governance debt. The decision point is whether the workflow can prove that its activity was worth the cost and risk.

Technical breakdown

Why token spend and value diverge in agentic workflows

Token spend measures model activity, not business effect. As agents take on larger tasks, they often consume more context, call more tools, and run more substeps before any useful output is accepted. That means cost can rise even when model prices fall, because the unit of work has changed from a single prompt to a multi-step execution path. In governance terms, the metering model is counting inputs to automation, not the actual control objective the automation was meant to achieve.

Practical implication: Treat token spend as an operational cost signal, not a success metric, when reviewing AI agent programmes.

What cost per outcome measures in AI operations

Cost per outcome divides total AI spend by a measured result such as an accepted pull request, a resolved ticket, a passing test, or a shipped feature. The metric only works when the outcome is observable and tied to a traceable run, which is why agent observability and evaluation are part of the measurement stack. This shifts reporting from volume to effectiveness and makes it possible to compare models, workflows, and use cases on a like-for-like basis.

Practical implication: Define outcome labels before scaling an AI workflow so each run can be measured against a business result.

Why traces and evals are the control layer behind the metric

Without traces, spend cannot be tied to a specific task or run. Without evals, success is only assumed, which makes cost per outcome impossible to trust. A trace shows what the agent did, while an evaluation shows whether the result met the task objective. Together they create an audit trail that supports both optimisation and governance, especially where AI agents act through tools and can cause downstream changes that are not visible in token billing alone.

Practical implication: Capture traces and attach evaluations before using cost per outcome in budgeting, review, or access decisions.

NHI Mgmt Group analysis

Cost per outcome is the right control variable for agentic AI governance. Token spend is too detached from business effect to support meaningful oversight once agents are running multi-step work. The real question is whether an automated action produced accepted output, not whether it consumed fewer tokens than last quarter. Organisations that keep reporting on spend alone will overestimate efficiency and under-govern AI access.

Outcome governance creates the missing bridge between AI performance and identity control. When an agent can open tools, move data, or influence code, the governance question becomes whether that access produced a verifiable result. That is where AI governance and identity security intersect: the agent’s permissions should be justified by measured outcomes, not by assumed productivity. Practitioners should treat access and spend as linked governance problems.

Traceability is the named concept this market still lacks: traced work that can be tied to a result. Cost per outcome only exists when every agent run is observable, attributable, and evaluable end to end. Without that, teams are left with budget caps and usage charts that cannot distinguish useful work from waste. The practitioner conclusion is simple: if the run cannot be traced to a result, it cannot justify itself.

AI budgets will keep misfiring until teams stop equating cheaper tokens with better control. Falling per-token prices can coexist with rising total cost because agent sessions expand and outcomes remain unmeasured. That means finance and security teams need a shared metric that reflects delivery, risk, and efficiency together. The operational conclusion is to govern AI on accepted work, not on consumed tokens.

From our research library:

• Claude Code-assisted commits leaked secrets at a rate of 3.2%, more than double the human-only baseline of 1.5%, with peaks reaching 31 secrets per 1,000 commits in August 2025, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Traceability will become the deciding control for agentic AI programmes. Teams that cannot connect agent runs to accepted outcomes will keep mistaking activity for value, and that creates a budget and governance blind spot. The practical next step is to move measurement from token volume to outcome evidence before the tool estate grows further.

Claude Code-assisted commits leaked secrets at a rate of 3.2%, more than double the human-only baseline of 1.5%, with peaks reaching 31 secrets per 1,000 commits in August 2025. That kind of operational signal shows why agent work cannot be judged by spend alone, because high-volume automation can simultaneously create output and expose control weaknesses.

For practitioners

• Define measurable outcomes for each AI use case Map every agent workflow to a business result such as resolved ticket, accepted change, passing test, or shipped feature before comparing spend across use cases.
• Capture traces for every agent run Record the prompt, tool calls, intermediate steps, and final output so spend can be tied to a specific execution rather than a monthly billing total.
• Attach evaluations to success criteria Score each run against a defined outcome so teams can distinguish accepted work from activity that only looked productive.
• Use cost per outcome in budget reviews Compare models and workflows by dollars per successful result, then redirect spend away from high-volume paths that do not produce accepted work.

Key takeaways

• AI token billing is a weak proxy for value once agents are doing multi-step work, because cheaper tokens can still produce more total spend and no better business result.
• Outcome-based measurement shifts governance from usage volume to accepted work, which is the only way to compare models and workflows fairly.
• Traceability and evaluation are the controls that make cost per outcome usable, especially when AI agents can influence code, tickets, or data.

Key terms

• Cost per outcome: A measure of the full cost required to complete a business task, including retries, review, exceptions, and rework. It is more useful than token cost alone because it reflects whether an AI workflow is actually efficient at scale.
• Agent Trace: A structured record of an AI agent’s runtime activity, including model calls, tool calls, approvals, and subagent steps. In practice, traces support debugging, evaluation, and governance when they are retained, searchable, and tied to the permissions behind the agent.
• Outcome Evaluation: The process of scoring an AI run against a defined success condition, such as an accepted change or resolved ticket. It turns subjective impressions of usefulness into evidence that can support budgeting, governance, and control decisions.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect access, lifecycle control, and governance decisions across modern identity programmes.