LLM token tracking reveals where production behavior breaks down

At a glance

What this is: This guide shows that aggregate LLM token totals obscure the real production failure mode, and that useful tracking must separate prompt, completion, context-window, and per-step usage.

Why it matters: IAM and AI security teams need this because token telemetry is only operationally meaningful when it can explain which call, span, or agent step drove the behaviour change.

Context

LLM token usage is a governance and observability problem, not just a billing metric. A total monthly or daily count can show spend trends, but it cannot tell whether the root cause is prompt growth, context-window pressure, or an agent retrying itself into waste. For production teams, the real question is where behaviour breaks down inside the request path.

That distinction matters because LLM systems now combine prompts, retrieval, tool calls, and agent steps. When usage is not broken out by call, request, and span, teams lose the ability to connect operational drift to the specific control that failed. In practice, token tracking becomes part of the AI control plane, and the same logic applies when AI agents consume non-human identity style credentials or drive repeated tool execution.

Key questions

Q: How should teams track LLM token usage when prompts, retrieval, and agent steps all contribute?

A: Track usage at three levels: per call, per request, and per span. Separate prompt and completion tokens, then attach those fields to traces so the team can see whether the problem is input growth, output growth, or repeated agent work. Without that structure, total usage hides the real production failure.

Q: Why do token totals fail to explain production latency and cost spikes?

A: Because the same spike can come from very different causes. Prompt bloat raises input processing time, context pressure pushes requests toward model limits, and agent retries or loops add hidden work deep in the trace. A rollup total shows that something changed, but not which control, step, or route caused it.

Q: What are the signs that LLM token tracking is failing as an operational control?

A: The main signs are unexplained cost spikes, rising p95 latency without a visible prompt change, and trace views that cannot identify the expensive step. If teams cannot separate prompt, completion, and span-level usage, the telemetry is too coarse to support investigation or governance.

Q: What is the difference between prompt-token tracking and span-level token tracking?

A: Prompt-token tracking tells you how much input a call consumed, while span-level tracking shows which step in the workflow burned the tokens. The first is useful for prompt design and context sizing. The second is essential for agent traces, where retries, tool calls, and sub-agent loops can dominate cost.

Technical breakdown

Why aggregate token totals hide production failure modes

A single token total collapses multiple different failure patterns into one number. Prompt bloat increases input volume, context-window pressure pushes requests toward the model limit, and agent loops burn tokens in repeated steps or retries. Those are distinct mechanisms with different fixes. If teams only watch rollup metrics, they cannot tell whether the problem sits in prompt design, retrieval sizing, or agent orchestration. The correct diagnostic unit is the smallest production boundary that still carries meaning: the call, the request, and the span. That is why token telemetry must be tied to traces rather than treated as a standalone cost feed.

Practical implication: Instrument usage at the call, request, and span level so the team can isolate which control actually failed.

How per-call prompt and completion tracking supports debugging

Prompt and completion tokens tell different stories. Prompt-heavy workloads usually point to expanding instructions, large tool schemas, retrieval overload, or long conversation history. Completion-heavy workloads often point to missing stop conditions, unconstrained generation, or reasoning output that expands unexpectedly. Splitting the fields lets teams separate latency before the first token from total response time, which matters when deciding whether the fix belongs in input shaping or output limits. This is also where OpenTelemetry GenAI attributes and provider usage payloads matter, because they preserve the split without forcing manual reconstruction from raw logs.

Practical implication: Track prompt and completion tokens separately so input growth and output growth do not get misdiagnosed as the same problem.

Why agent spans need token attribution

Agentic workflows create hidden cost centres because the expensive step is often not the top-level answer. A tool call, retrieval step, retry cluster, or sub-agent loop can consume far more tokens than the visible response. Span-level attribution exposes that structure by attaching usage to each step in the trace tree. Once that exists, teams can sort by token count, cost, or latency and identify which span is inflating the run. Without span attribution, the dashboard shows that the agent was expensive, but not why. That is a governance gap as much as an observability gap, because it prevents accountable control over autonomous behaviour.

Practical implication: Attach token usage to each agent span so expensive retries and tool loops are visible before they become budget or latency incidents.

Threat narrative

Attacker objective: The objective is to consume compute and obscure the true failure point so the organisation cannot rapidly distinguish benign growth from runaway production behaviour.

1. Entry occurs through an LLM request whose aggregate token total looks normal even though the prompt or retrieval payload has already grown beyond a safe operating shape.
2. Credential or control abuse appears when repeated tool calls, retries, or sub-agent loops consume tokens inside the trace without being obvious in top-line reporting.
3. Impact is production drift, where teams lose the ability to distinguish prompt bloat, context pressure, and runaway orchestration before cost and latency become operational incidents.

Breaches seen in the wild

• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Token telemetry is becoming a governance control, not a reporting convenience. Once LLM systems are embedded in production workflows, aggregate usage totals are too blunt to support decision-making. Teams need token data attached to traces, prompts, and spans so they can explain behaviour, not just count it. The practical implication is that observability and governance have to be designed together.

Prompt bloat is a control drift problem, not simply an efficiency issue. Small template edits, larger tool descriptions, and longer fallback instructions accumulate into persistent input growth. That means the relevant governance question is not whether a model is expensive, but which change introduced the drift and how quickly it can be linked to a prompt version. Practitioners should treat prompt versioning as a control surface.

Context-window pressure creates a distinct failure mode that generic token totals cannot reveal. Requests near the model limit behave differently from ordinary high-volume traffic, especially in RAG and chat systems where history and retrieval compete for space. The control gap here is visibility at the request level, because median usage can look safe while a narrow set of tenants or routes is already operating at the edge. The implication is that teams need utilization thresholds by tenant, feature, and route.

Agent span attribution is the named concept that matters here: hidden token debt. That debt accumulates when retries, tool calls, and sub-agent loops burn tokens deep inside traces while the top-level run appears normal. In AI governance terms, this is a traceability problem because the cost signal and the behavioural signal diverge. Practitioners should make span-level token attribution mandatory wherever autonomous steps can multiply work.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Hidden token debt: production teams should treat unexplained token growth inside traces as a traceability failure, not merely a budgeting issue. When agent loops, retries, and retrieval steps are not visible at span level, governance moves too late to matter.

Token telemetry becomes operationally useful only when teams can answer the control question at the right boundary: which call, which request, and which step changed behaviour. That is the point at which observability turns into enforcement, because drift can be tied back to a specific prompt version or orchestration path.

For practitioners

• Implement token telemetry at the trace boundary Attach prompt, completion, and reasoning-token usage to every LLM call so the operational record explains the request path, not just the monthly total.
• Separate prompt and completion metrics Report input and output tokens independently, then compare them across prompt versions, models, and routes to identify whether input growth or output growth is driving the spike.
• Set context utilization thresholds Calculate request-level context-window utilisation as a percentage and alert on p95 pressure by tenant, feature, model, and route before overflow errors appear.
• Instrument agent spans for token attribution Capture token counts on tool calls, retrieval spans, retries, and sub-agent steps so the expensive step is visible inside the trace tree.
• Version prompts as a change-controlled asset Treat prompt edits, tool schema growth, and fallback instructions as controlled changes so token drift can be correlated to a specific revision.

Key takeaways

• Aggregate token totals are too coarse to explain why production LLM systems slow down or become expensive.
• The useful control is not a single count but a layered view of per-call usage, request-level context pressure, and span-level agent activity.
• Teams that cannot attribute token spikes to a specific trace step will struggle to govern prompt changes, retries, and runaway loops.

Key terms

• Token Telemetry: Token telemetry is the measurement of how many tokens an AI system consumes, by whom, and for what workload. In practice it gives security, platform, and finance teams the data they need to enforce budgets, detect abuse, and attribute AI usage to the right identity.
• Context Window: The context window is the text a model receives at one time, including prompts, retrieved documents, and conversation history. Security teams care about it because it becomes the practical boundary between trusted instructions and untrusted content, especially when the application assembles that text automatically.
• Span-Level Attribution: Span-level attribution is the assignment of token usage and other runtime data to each step inside an agent or request trace. It is the mechanism that shows which tool call, retrieval step, retry, or sub-agent run actually generated the cost or latency.
• Answer Drift: Answer drift is the gradual change in a model’s responses over time, often showing up as reduced consistency or increasing error rates. It can signal degraded grounding, shifting data quality, or prompt and retrieval issues. Monitoring drift helps teams catch reliability problems before they become widespread user-facing failures.

Deepen your knowledge

NHI Mgmt Group’s NHI Foundation Level course covers NHI governance, machine identity security, and the control logic that underpins secure access to AI systems. It helps security practitioners connect identity governance to the broader programmes they run across cloud, data, and agentic workloads.