Topic starter
18/08/2026 2:32 pm
TL;DR: AI overspend is driven by agent behaviour, not just model pricing, and Fiddler argues that request-level and span-level telemetry are needed to trace cost back to the prompt, workflow, and retry loop that caused it. Visibility alone is insufficient because budgets must be enforced inline on the request path, not after the invoice arrives.
Editorial analysis by NHI Mgmt Group, based on content published by Fiddler: “AI Cost Tracking: How to Trace Overspend to the Request That Caused It”.
Key questions
Q: What breaks when AI cost is tracked only at the invoice level?
A: Invoice-level tracking hides the request, retry loop, and agent step that created the overspend.
Q: Why do request-level budgets matter more than monthly AI spend reviews?
A: Monthly reviews arrive after the expensive behaviour has already repeated many times, so they are too late to change the outcome.
Q: What should teams do when AI cost attribution is unclear?
A: They should reconstruct the run from telemetry before changing policy.
Practitioner guidance
- Instrument request-level telemetry on every AI workflow Capture provider, model, prompt, run_id, step_name, retry_count, workflow, team, tenant, and estimated_cost on each span so spend can be traced to the exact request path.
- Separate visibility from enforcement in your control design Use dashboards for reporting, but enforce spend limits inline on the request path so a runaway loop can be constrained before it generates another costly call.
- Reconcile calculated cost against provider bills regularly Compare your attributed cost totals with the provider invoice to catch stale price tables, cache effects, or untagged usage that would otherwise distort governance reporting.
Bottom line: AI cost overruns are produced by runtime behaviour such as retries, context growth, and agent loops, so invoice totals alone cannot explain them.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
18/08/2026 3:19 pm
AI cost attribution debt: when teams can only see account totals, they accumulate governance debt that hides the operational source of spend. Billing dashboards were never designed to explain runtime lineage across prompts, tools, retries, and evaluation loops. The practical conclusion is that AI governance has to move from finance summaries to request-level accountability.
A question worth separating out:
Q: What should security and platform teams do when evaluation traffic becomes a major cost driver?
A: Treat evaluation as part of the governed AI workload, not as invisible test traffic. Track judge-model calls, assign ownership, and apply the same tracing and policy controls used for production inference. If evaluation runs through external APIs without attribution, it can become a second cost centre that never appears clearly in operational reporting.
👉 Read our full editorial: AI cost tracking needs request-level attribution to stop overspend