Topic starter
19/08/2026 3:48 pm
TL;DR: AI token prices are falling, but usage is rising faster, so token spend no longer tells teams whether AI is producing value; Arize argues for measuring cost per outcome instead, using traced and evaluated agent runs to connect spend to resolved tickets, accepted PRs, or shipped features. The practical shift is from billing visibility to outcome governance, where cost efficiency is judged by work actually completed rather than tokens consumed.
Editorial analysis by NHI Mgmt Group, based on content published by Arize: “Why AI token costs don’t tell you if your AI is working”.
Key questions
Q: How should teams measure whether AI agents are actually delivering value?
A: Teams should measure cost per outcome, not just token consumption.
Q: Why does lower token pricing not prove that AI is becoming more efficient?
A: Lower token pricing only shows that a unit of model output is cheaper.
Q: What breaks when AI traces are kept separate from evaluation workflows?
A: Iteration slows down because teams have to export data manually, rebuild context in other tools, and lose the feedback loop between production behavior and quality improvement.
Practitioner guidance
- Define measurable outcomes for each AI use case Map every agent workflow to a business result such as resolved ticket, accepted change, passing test, or shipped feature before comparing spend across use cases.
- Capture traces for every agent run Record the prompt, tool calls, intermediate steps, and final output so spend can be tied to a specific execution rather than a monthly billing total.
- Attach evaluations to success criteria Score each run against a defined outcome so teams can distinguish accepted work from activity that only looked productive.
Bottom line: AI token billing is a weak proxy for value once agents are doing multi-step work, because cheaper tokens can still produce more total spend and no better business result.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
19/08/2026 4:24 pm
Cost per outcome is the right control variable for agentic AI governance. Token spend is too detached from business effect to support meaningful oversight once agents are running multi-step work. The real question is whether an automated action produced accepted output, not whether it consumed fewer tokens than last quarter. Organisations that keep reporting on spend alone will overestimate efficiency and under-govern AI access.
A few things that frame the scale:
- Claude Code-assisted commits leaked secrets at a rate of 3.2%, more than double the human-only baseline of 1.5%, with peaks reaching 31 secrets per 1,000 commits in August 2025, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: What should identity and AI governance teams do when agent access is hard to justify by output?
A: They should treat access as conditional on measurable results. If an agent cannot demonstrate accepted work, its permissions, budget, and scope should be reviewed together, because unchecked access with no outcome evidence creates governance debt. The decision point is whether the workflow can prove that its activity was worth the cost and risk.
👉 Read our full editorial: AI token spend obscures whether agents are delivering business value