Notifications
Clear all
Topic starter
19/08/2026 3:45 pm
TL;DR: High token spend comes from three separate forces, inference waste, evaluation overhead, and incident exposure from unmonitored traces, according to Fiddler, while enterprises already spent $8.4 billion on LLM APIs in the first half of 2025. The real issue is attribution: without per-agent visibility, organisations cannot control cost, risk, or governance as agentic systems scale.
NHIMG editorial — based on content published by Fiddler: How High Token Costs Quietly Erode AI ROI
By the numbers:
- Enterprise LLM API spending hit $8.4 billion in the first half of 2025, showing how quickly production AI costs are scaling.
- The Soo Group documented a case where monthly AI operating costs rose from $1,500 in POC to just over $1 million in production, a 717x increase.
- In one deployment, the LLM API line item jumped from $500 to $847,293, roughly a 1,700x increase on its own.
Questions worth separating out
Q: How should security teams govern token costs in production AI systems?
A: Start with per-agent attribution so every call can be linked to a workflow, policy, or business function.
Q: Why do agentic AI systems create hidden cost and risk exposure?
A: Because one user request can fan out into multiple model calls, evaluations, and tool invocations that are invisible in aggregated billing.
Q: What do organisations get wrong about AI observability?
A: They often confuse technical telemetry with governance evidence.
Practitioner guidance
- Implement per-agent token attribution Capture model, input tokens, output tokens, cost, and originating agent on every request so spend can be tied to a specific workflow or policy violation.
- Separate inference and evaluation telemetry Tag evaluation calls independently from user-facing inference calls so external monitoring costs do not disappear into one aggregated bill.
- Set guardrails on context growth and retries Cap prompt size, retrieval volume, and repeated tool-call loops to stop runaway token burn in multi-agent workflows.
What's in the full article
Fiddler's full blog post covers the operational detail this post intentionally leaves for the source:
- Per-call telemetry patterns for capturing model, token, latency, and estimated cost at the agent level.
- The practical differences between inference spend, evaluation spend, and hidden monitoring overhead.
- Example instrumentation code for tracking token usage across OpenAI-compatible responses.
- The specific failure modes that make context inflation, retries, and sampling more expensive in production than in pilot environments.
👉 Read Fiddler's analysis of how token costs erode AI ROI →
Token costs and AI ROI: where do enterprise controls break down?
Explore further
19/08/2026 4:23 pm
Token governance is becoming a control discipline, not a finance exercise. Once AI systems operate at production scale, the question is no longer only what they cost, but whether each token-consuming action is attributable to a specific agent, workflow, or policy choice. That puts AI observability, policy enforcement, and cost attribution into the same governance conversation. For IAM and AI security teams, the practitioner conclusion is straightforward: if you cannot attribute behaviour, you cannot govern it.
A few things that frame the scale:
- Enterprise LLM API spending hit $8.4 billion in the first half of 2025, according to LLMjacking: How Attackers Hijack AI Using Compromised NHIs.
- 64% of valid secrets leaked in 2022 are still valid and exploitable today, according to The State of Secrets Sprawl 2026.
A question worth separating out:
Q: How do you know if AI trace sampling is creating a governance gap?
A: You know it is creating a gap when the system's highest-risk actions are covered by only a fraction of traces while cost reports still look healthy. If outputs can affect customers, compliance, or tool execution, sampled monitoring is not enough. High-risk paths need complete coverage and alerting, not statistical comfort.
👉 Read our full editorial: Token governance is now central to AI ROI and agentic scale
