AI cost tracking needs request-level attribution to stop overspend

At a glance

What this is: This article argues that AI overspend is a traceability problem, not a billing problem, and that request-level and span-level telemetry are required to explain variable token cost.

Why it matters: For IAM, NHI, and AI governance teams, the same lineage discipline used to control privileged access now needs to extend to AI requests, agent steps, and owners.

By the numbers:

• An AI bill can jump 40% while traffic stays flat, according to Fiddler.
• Token usage for the same LLM agent task can vary by up to 30 times across runs, according to Fiddler's cited research.

👉 Read Fiddler's analysis of AI cost tracking and request-level overspend attribution

Context

AI cost tracking becomes necessary when a variable runtime behaviour changes spend faster than finance teams can explain it. The first problem is attribution: an invoice can show the total, but it cannot tell you which request, agent step, or retry loop created the overrun. That matters because the same lineage gap appears wherever AI systems consume model tokens, secrets, or privileged access as part of runtime execution.

For identity and governance teams, this is a familiar control problem in a new layer. If you cannot attribute action to an owner and a step, you cannot govern it effectively. The article frames AI spend as an operational signal, which is consistent with how NHI and agentic AI programmes need to treat runtime behaviour: measurable, attributable, and enforceable.

Key questions

Q: What breaks when AI cost is tracked only at the invoice level?

A: Invoice-level tracking hides the request, retry loop, and agent step that created the overspend. It also collapses different workflows into a single total, which makes ownership unclear and slows remediation. Teams end up explaining the bill after the fact instead of controlling the behaviour that generated it.

Q: Why do agent retries and context growth make AI spend hard to predict?

A: Retries resend context, and context grows as tool outputs accumulate across the run. That means the same task can consume very different token volumes depending on how often the agent loops, how many tools it calls, and how much prior output it carries forward.

Q: How can security teams tell whether AI lifecycle controls are working?

A: They should look for evidence that access requests, policy enforcement, and usage visibility are centrally recorded and current. If those signals are fragmented across platforms, the programme may be documenting governance rather than enforcing it. Continuous traceability is the practical test.

Q: How should organisations enforce AI budgets without waiting for monthly reports?

A: Use inline policy at the gateway or request layer so a workflow can be slowed, blocked, or redacted before the model call completes. That turns budget rules into runtime governance rather than retrospective finance review.

Technical breakdown

Why request-level cost attribution matters

AI spend is not fixed in the way seat-based software cost is fixed. Token usage changes with prompt length, tool calls, retries, and context growth, so the same workflow can produce very different bills across runs. That makes provider totals too coarse to explain variance. Request-level attribution ties each token to the exact request, model, and owner, which turns billing from a monthly summary into an operational record. In identity terms, this is lineage control for AI execution: who did what, when, through which path, and under whose policy.

Practical implication: instrument every high-value workflow so cost can be traced to a single request and owner.

How span-level telemetry exposes hidden cost drivers

A span is the smallest useful unit for understanding where cost is created, such as one model call or one tool invocation. When spans roll up into traces, sessions, and application views, teams can see whether retries, context accumulation, or evaluation traffic caused the spike. This is the same architectural logic used in observability and audit trails: aggregate views are useful, but only when they can be drilled down to the event that created them. Without span-level detail, expensive loops look like normal usage.

Practical implication: capture cost on every span so retries and agent loops can be isolated before they spread.

Why inline enforcement beats after-the-fact reconciliation

Observing spend tells you what already happened. Enforcing budgets inline changes what can happen next. In AI systems, that means policies must act on the request path before the model call completes, rather than waiting for an invoice or a dashboard alert. This is especially relevant where agents can chain decisions, because the expensive behaviour may complete before a human review cycle begins. From a governance perspective, cost control becomes a runtime policy problem, not a finance reconciliation task.

Practical implication: connect budget thresholds to request-path policy so runaway behaviour is blocked in session, not discovered later.

Threat narrative

Attacker objective: The objective is not theft but uncontrolled consumption, where opaque agent behaviour drives avoidable cost and weakens operational governance.

1. Entry occurs when an AI workflow or agent begins issuing requests whose token usage is not fully attributed to a specific span or owner.
2. Escalation happens as retries, context growth, and nested tool calls multiply token consumption across the same session without hard runtime limits.
3. Impact is spend inflation that cannot be explained from the invoice alone, delaying remediation and obscuring the workflow that caused the overspend.

NHI Mgmt Group analysis

AI cost governance is becoming a lineage problem before it is a finance problem. The article shows that spending variance comes from runtime behaviour, which means the control issue is attribution, not simply price. That same logic applies to IAM and NHI programmes: if a request, token, or agent step cannot be tied to an owner and policy, the organisation cannot govern the outcome. Practitioners should treat cost lineage as part of identity lineage, not as a separate accounting function.

Request-level telemetry is the named control gap this article exposes. The useful concept here is cost attribution gap, meaning the distance between a runtime action and the business owner who can explain it. Provider dashboards collapse that gap into totals, but total-only visibility is too blunt for AI governance. Teams need per-request lineage, span-level detail, and owner metadata if they want to prevent repeated overspend. Practitioners should design telemetry with auditability in mind from the start.

Inline enforcement is the only control that can stop agentic overspend in time. The article correctly separates observability from control, because alerts that arrive after the bill is already generated are operational reporting, not governance. In agentic systems, the risk is magnified by chained decisions and retries that can complete before any review. Practitioners should assume that runtime policy is mandatory wherever an AI system can recurse, retry, or invoke tools on its own.

AI governance needs to absorb the same discipline that identity teams already use for privileged activity. A model, workflow, and agent step should not be treated as anonymous compute. They need ownership, policy boundaries, and measurable boundaries for cost as much as for access. That is especially relevant for NHI and agentic AI programmes, where the line between a tool call and a governed identity event is disappearing. Practitioners should align AI observability with identity governance before usage scales further.

Named concept: the cost attribution gap. This article gives a precise label to a common failure mode in AI operations, where spending can be measured but not explained. Once that gap exists, finance sees variance but engineering cannot name the cause quickly enough to intervene. Practitioners should close the gap with request-level lineage, ownership tags, and enforcement hooks at the gateway.

What this signals

Cost attribution gap: once AI systems start chaining retries, tool calls, and evaluations, finance and security teams will need a shared runtime view of who owns each expensive action. That view should be built around identity lineage, not just billing metadata.

Practitioners should expect agentic systems to surface the same governance pressure that NHIs already create in secrets management: if an action cannot be attributed and enforced at runtime, it will eventually appear as cost, risk, or both. Teams that already use strong identity controls have a head start because the operating model is similar: define owner, bound scope, and verify execution path.

For practitioners

• Instrument request-level lineage across AI workflows Capture the model, prompt, agent, workflow, run ID, step name, retry count, and owner on every span so a single cost spike can be traced to one decision path.
• Reconcile span cost against provider invoices regularly Compare calculated span-level spend with the monthly provider bill to surface stale pricing tables, cache discounts, and untagged usage before they distort reporting.
• Enforce budgets at the request path Apply allow, block, or redact decisions inline when a workflow exceeds its cost threshold so runaway loops are stopped before they complete the session.
• Separate evaluation traffic from production inference Tag judge-model calls and QA runs independently so evaluation spend does not hide inside primary inference totals or distort unit economics.

Key takeaways

• AI overspend is a governance problem created by runtime behaviour, not a billing problem caused by pricing alone.
• Request-level and span-level telemetry are required to explain which workflow, agent step, or retry loop drove the spike.
• Inline policy enforcement is the control that stops runaway spend before the invoice turns it into a retrospective exercise.

Key terms

• Cost Attribution: Cost attribution is the practice of assigning spend to the specific request, workflow, owner, or system action that created it. In AI operations, it goes beyond totals and links token consumption to execution lineage so teams can explain variance and enforce accountability.
• Span-Level Telemetry: Span-level telemetry records the smallest meaningful unit of work inside an AI run, such as one model call or tool invocation. It lets teams trace cost, retries, and behaviour back to the exact step that created them instead of relying on aggregate totals.
• Agentic Hierarchy: An agentic hierarchy is the layered structure of AI execution, from individual spans to traces, sessions, and application views. It matters because cost, risk, and policy decisions often happen at the span level, while governance decisions are made higher up the stack.
• Inline Enforcement: Inline enforcement is the technical act of applying access policy in the live session path, not just at approval time. It matters because identity governance without runtime enforcement can authorize access that the session layer never actually constrains, especially in distributed and third-party environments.

What's in the full article

Fiddler's full blog covers the operational detail this post intentionally leaves for the source:

• The exact OpenTelemetry span attributes used to trace token cost back to a request, workflow, and owner.
• The implementation pattern for server-side pricing maps and how reconciliation against provider invoices is handled.
• Examples of inline allow, block, and redact enforcement at the gateway layer for AI requests.
• The distinction between evaluation cost, inference cost, and observability overhead in production AI systems.

👉 Fiddler's full blog covers the tracing model, metadata fields, and inline enforcement approach in more implementation detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and agentic AI identity. It helps practitioners connect runtime behaviour to the governance controls their programmes need.