Notifications
Clear all
Topic starter
19/08/2026 3:48 pm
TL;DR: AI token prices are falling, but usage is rising faster, so token spend no longer tells teams whether AI is producing value; Arize argues for measuring cost per outcome instead, using traced and evaluated agent runs to connect spend to resolved tickets, accepted PRs, or shipped features. The practical shift is from billing visibility to outcome governance, where cost efficiency is judged by work actually completed rather than tokens consumed.
NHIMG editorial — based on content published by Arize: Why AI token costs don’t tell you if your AI is working
Questions worth separating out
Q: How should teams measure whether an AI workflow is actually working?
A: Measure AI cost against a verified outcome such as a resolved ticket, accepted pull request, or shipped feature.
Q: Why do token metrics fail as a governance signal for AI systems?
A: Token metrics fail because they measure volume, not usefulness.
Q: What do security teams get wrong about AI cost control?
A: They often treat cost as a finance-only issue and overlook the identity layer that drives usage.
Practitioner guidance
- Define outcome metrics before expanding AI usage Choose one or two business outcomes per workflow, such as resolved tickets, accepted PRs, or shipped features, and make them the denominator for spend analysis.
- Instrument every agent run with traceability Capture prompts, tool calls, and permissioned actions so each run can be linked to a specific task and reviewed against its result.
- Separate value review from usage review Review token growth, outcome success, and access scope as different controls so a high-usage system is not automatically treated as a successful one.
What's in the full article
Arize's full article covers the operational detail this post intentionally leaves for the source:
- The cost-per-outcome implementation logic behind trace capture and evaluation scoring for AI workflows.
- Examples of how teams can map spend to resolved tickets, accepted pull requests, and shipped features.
- The observability workflow used to turn agent runs into auditable performance and budget data.
- The practical distinction between token-based billing, outcome-based pricing, and outcome-based governance.
👉 Read Arize's analysis of why token costs do not show whether AI is working →
AI token spend vs outcomes: what should teams measure instead?
Explore further
19/08/2026 4:24 pm
Token spend creates governance blindness when it is treated as a proxy for value. AI systems can consume large volumes of compute, prompts, and delegated access while producing no measurable business outcome. That is not just a finance problem, because access and runtime permissions are being exercised inside systems whose usefulness has not been proven. The result is a control gap where activity is mistaken for effectiveness, and effectiveness is what justifies privilege. Practitioners should treat token cost as an accounting input, not an assurance signal.
A question worth separating out:
Q: Who should own cost per outcome reporting for AI programmes?
A: Ownership should sit across finance, engineering, and security, because the metric joins spend, delivery, and control effectiveness. Security and identity teams should ensure the system of record includes traces and access context, while product or engineering teams validate the business result.
👉 Read our full editorial: AI token spend obscures whether agents are delivering business value
