AI cost tracking needs request-level attribution to stop overspend

At a glance

What this is: Fiddler argues that AI cost control fails when teams look only at invoices, because overspend is produced by request-level runtime behaviour, not just model pricing.

Why it matters: For AI governance and IAM teams, cost attribution has to follow the request path and ownership chain so budgets, policy, and accountability can be enforced where spend is created.

Context

AI cost governance breaks when teams treat monthly bills as the primary control point. In production AI systems, spend is created by requests, retries, context growth, and tool use, so the useful unit of analysis is the span or request rather than the invoice total. That makes cost an operational governance problem, not a finance-only one.

The article’s core identity angle is not human IAM but AI workflow ownership and runtime lineage. Each request needs enough metadata to tie spend back to the model, prompt, agent, workflow, tenant, and owner, otherwise neither accountability nor control can be enforced with confidence.

Key questions

Q: What breaks when AI cost is tracked only at the invoice level?

A: Invoice-level tracking hides the request, retry loop, and agent step that created the overspend. It also collapses different workflows into a single total, which makes ownership unclear and slows remediation. Teams end up explaining the bill after the fact instead of controlling the behaviour that generated it.

Q: Why do request-level budgets matter more than monthly AI spend reviews?

A: Monthly reviews arrive after the expensive behaviour has already repeated many times, so they are too late to change the outcome. Request-level budgets matter because they can stop or constrain the next costly call while the workflow is still running. That is the difference between governance and post-event accounting.

Q: What should teams do when AI cost attribution is unclear?

A: They should reconstruct the run from telemetry before changing policy. Join gateway logs, agent traces, model pricing, and outcome data so you can see which span, workflow, or tenant created the overage. Without that attribution, cost controls will be too blunt to fix the actual issue.

Q: What should security and platform teams do when evaluation traffic becomes a major cost driver?

A: Treat evaluation as part of the governed AI workload, not as invisible test traffic. Track judge-model calls, assign ownership, and apply the same tracing and policy controls used for production inference. If evaluation runs through external APIs without attribution, it can become a second cost centre that never appears clearly in operational reporting.

Technical breakdown

Why token-based AI costs are non-deterministic

Token pricing looks simple on paper, but production cost is shaped by runtime behaviour. A request may trigger tool calls, retries, or longer context windows, which makes token consumption vary from run to run even when traffic volume is flat. That means the same business workflow can produce very different cost outcomes depending on agent behaviour, prompt length, and reprocessing loops. Aggregate billing hides those mechanics, so finance totals do not explain operational causality. The technical issue is not just price per token, but variable token generation across the lifecycle of a single request.

Practical implication: Treat cost as a runtime control problem and instrument every request where token growth can occur.

Why span-level tracing is the right unit for AI cost attribution

A span captures one model call or tool invocation, which is the point where spend is actually incurred. When spans roll up into traces, sessions, and application views, teams can move from a large unexplained bill to a specific retry loop, prompt revision, or agent step. This is the difference between visibility and attribution: one shows that spend rose, the other shows why. Without span lineage, orchestration, retrieval, and evaluation costs remain buried inside aggregate totals and cannot be assigned to a clear owner or workflow.

Practical implication: Attach cost metadata at span creation so each expensive step can be traced back to the exact request path.

What inline enforcement changes in AI governance

Observability alone only tells you what already happened. Inline enforcement acts on the request before the next expensive call is executed, which is the only point where budget policy can actually stop runaway behaviour. In governance terms, this moves cost control from after-the-fact reconciliation into live decisioning on the request path. That matters when loops, retries, or evaluation traffic can multiply cost faster than a monthly review cycle can react. The control plane must sit close to the agent runtime, not in a spreadsheet after the invoice arrives.

Practical implication: Use request-path policy to block or constrain expensive runs before they compound into uncontrolled spend.

NHI Mgmt Group analysis

AI cost attribution debt: when teams can only see account totals, they accumulate governance debt that hides the operational source of spend. Billing dashboards were never designed to explain runtime lineage across prompts, tools, retries, and evaluation loops. The practical conclusion is that AI governance has to move from finance summaries to request-level accountability.

Cost visibility is not cost control: a dashboard can show drift, but it cannot prevent the next expensive request. That distinction matters because many AI programmes still mistake observability for enforcement. The control point has to sit on the request path if budgets are meant to mean anything operationally.

Request lineage is now part of AI identity governance: the article shows that every AI workflow needs owner, tenant, and runtime metadata to make spend attributable. In effect, the AI system becomes governable only when the request can be tied to a named workflow and accountable business context. For practitioners, the key question is whether attribution exists at the moment spend is created.

Evaluation traffic is a hidden governance surface: quality checks, judge models, and internal testing can quietly become part of the cost problem. That means AI assurance and AI cost governance are no longer separable disciplines. Practitioners should treat evaluation paths as production-like spend paths, with the same tracing and policy expectations.

What this signals

AI cost attribution debt: cost governance is drifting toward the same problem identity teams faced with shadow systems, where owners can see totals but cannot explain the individual unit of activity driving them. The practical response is to make request lineage, ownership, and policy enforcement part of the runtime design rather than a reporting afterthought.

The programme implication is straightforward: if an AI workflow can retry, expand context, or call tools without a traceable owner, it can also evade meaningful budget control. Teams should expect cost telemetry to become part of their governance baseline, not a specialised optimisation exercise.

For practitioners

• Instrument request-level telemetry on every AI workflow Capture provider, model, prompt, run_id, step_name, retry_count, workflow, team, tenant, and estimated_cost on each span so spend can be traced to the exact request path.
• Separate visibility from enforcement in your control design Use dashboards for reporting, but enforce spend limits inline on the request path so a runaway loop can be constrained before it generates another costly call.
• Reconcile calculated cost against provider bills regularly Compare your attributed cost totals with the provider invoice to catch stale price tables, cache effects, or untagged usage that would otherwise distort governance reporting.
• Tag every agent step with an accountable owner Make workflow, team, tenant, and environment mandatory fields so each expensive span can be assigned to a responsible business or technical owner.
• Keep evaluation traffic inside governed environments Track judge-model calls and internal test runs alongside production inference so quality assurance does not become an invisible second billing channel.

Key takeaways

• AI cost overruns are produced by runtime behaviour such as retries, context growth, and agent loops, so invoice totals alone cannot explain them.
• Request-level and span-level telemetry provide the operational lineage needed to identify the exact step, workflow, or owner behind a spend spike.
• Inline enforcement is the control difference that matters, because budgets only stop overspend when they act before the next costly request runs.

Key terms

• Session-Level Attribution: The ability to tie an action back to a specific runtime session, actor, and policy state. For AI agents, this matters because network logs alone often cannot show whether activity came from an approved workflow, a shadow tool, or a reused user entitlement.
• Span-Level Telemetry: Span-level telemetry records the smallest meaningful unit of work inside an AI run, such as one model call or tool invocation. It lets teams trace cost, retries, and behaviour back to the exact step that created them instead of relying on aggregate totals.
• Inline Enforcement: Inline enforcement is the technical act of applying access policy in the live session path, not just at approval time. It matters because identity governance without runtime enforcement can authorize access that the session layer never actually constrains, especially in distributed and third-party environments.
• Evaluation Traffic: Evaluation traffic is the token consumption created by automated quality checks, judge models, and testing workflows. It often sits alongside production inference and can become a major hidden cost if it is not traced, owned, and governed like any other AI workload.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and secrets management. It helps security and platform teams build accountability into the systems that now create and control access at runtime.