Notifications
Clear all
Topic starter
18/08/2026 2:32 pm
TL;DR: AI overspend is driven by agent behaviour, not just model pricing, and Fiddler argues that request-level and span-level telemetry are needed to trace cost back to the prompt, workflow, and retry loop that caused it. Visibility alone is insufficient because budgets must be enforced inline on the request path, not after the invoice arrives.
NHIMG editorial — based on content published by Fiddler: AI Cost Tracking: How to Trace Overspend to the Request That Caused It
By the numbers:
- An AI bill can jump 40% while traffic stays flat, according to Fiddler.
- Token usage for the same LLM agent task can vary by up to 30 times across runs, according to Fiddler's cited research.
Questions worth separating out
Q: What breaks when AI cost is tracked only at the invoice level?
A: Invoice-level tracking hides the request, retry loop, and agent step that created the overspend.
Q: Why do agent retries and context growth make AI spend hard to predict?
A: Retries resend context, and context grows as tool outputs accumulate across the run.
Q: How can security teams tell whether AI lifecycle controls are working?
A: They should look for evidence that access requests, policy enforcement, and usage visibility are centrally recorded and current.
Practitioner guidance
- Instrument request-level lineage across AI workflows Capture the model, prompt, agent, workflow, run ID, step name, retry count, and owner on every span so a single cost spike can be traced to one decision path.
- Reconcile span cost against provider invoices regularly Compare calculated span-level spend with the monthly provider bill to surface stale pricing tables, cache discounts, and untagged usage before they distort reporting.
- Enforce budgets at the request path Apply allow, block, or redact decisions inline when a workflow exceeds its cost threshold so runaway loops are stopped before they complete the session.
What's in the full article
Fiddler's full blog covers the operational detail this post intentionally leaves for the source:
- The exact OpenTelemetry span attributes used to trace token cost back to a request, workflow, and owner.
- The implementation pattern for server-side pricing maps and how reconciliation against provider invoices is handled.
- Examples of inline allow, block, and redact enforcement at the gateway layer for AI requests.
- The distinction between evaluation cost, inference cost, and observability overhead in production AI systems.
👉 Read Fiddler's analysis of AI cost tracking and request-level overspend attribution →
AI cost attribution gaps: what practitioners need to trace first?
Explore further
18/08/2026 3:19 pm
AI cost governance is becoming a lineage problem before it is a finance problem. The article shows that spending variance comes from runtime behaviour, which means the control issue is attribution, not simply price. That same logic applies to IAM and NHI programmes: if a request, token, or agent step cannot be tied to an owner and policy, the organisation cannot govern the outcome. Practitioners should treat cost lineage as part of identity lineage, not as a separate accounting function.
A question worth separating out:
Q: How should organisations enforce AI budgets without waiting for monthly reports?
A: Use inline policy at the gateway or request layer so a workflow can be slowed, blocked, or redacted before the model call completes. That turns budget rules into runtime governance rather than retrospective finance review.
👉 Read our full editorial: AI cost tracking needs request-level attribution to stop overspend
