LLM token tracking reveals where production behavior breaks down

At a glance

What this is: This is a practical guide to tracking LLM token usage at call, request, and span level, with the key finding that aggregate totals mask prompt bloat, context pressure, and agent loops.

Why it matters: It matters because IAM, NHI, and AI governance teams need per-step visibility to control how AI systems consume credentials, context, and cost across production workflows.

By the numbers:

• 28.65 million new hardcoded secrets were detected in public GitHub commits in 2025 alone, a 34% year-over-year increase and the largest single-year jump ever recorded.
• 64% of valid secrets leaked in 2022 are still valid and exploitable today, proving that detection alone is not enough without automated revocation.

👉 Read Braintrust's guide to tracking LLM token usage in production

Context

LLM token tracking is often treated as a billing concern, but in production it is really an observability problem. When prompt growth, context-window pressure, or agent retries drive token spikes, aggregate totals tell teams that something changed without showing where the workflow broke. That gap matters for AI governance because the same telemetry used for cost can also expose control failures in agent behaviour and request shaping.

For identity and access programmes, the governance question is not only what the model consumed, but which runtime steps and trusted inputs made that consumption possible. In environments where AI systems rely on service credentials, retrieval sources, or delegated tools, per-call and per-span visibility becomes part of NHI control, not just application telemetry. The operating pattern in the article is typical of teams that have outgrown dashboard totals but have not yet built request-level accountability.

One useful adjacent concept is token telemetry as a control surface. Once token counts are tied to traces, teams can evaluate whether prompts, retrieval, and agent loops are staying inside approved operating bounds rather than only watching monthly usage totals.

Key questions

Q: What breaks when LLM token usage is only tracked as a monthly total?

A: Aggregate totals hide whether the problem is prompt bloat, context pressure, or a runaway agent loop. Without per-call and per-span attribution, teams can see that usage increased but cannot tell which request, route, or step caused it. That makes debugging slow and can also mask governance failures in production AI workflows.

Q: How should security teams govern AI agent token spend without losing accountability?

A: Treat token spend as an identity control problem. Give each agent its own credential, scope its tool access tightly, and use session boundaries so every model call can be attributed to a specific executor and task. That makes cost governance, access review, and revocation part of the same control plane.

Q: How do teams know when context-window pressure is becoming a production risk?

A: Watch p95 utilisation by route, tenant, model, and feature rather than relying on average usage. If requests repeatedly approach the model limit, the workflow is likely to become brittle through overflow errors, truncated history, or expensive retries, and the fix usually requires prompt, retrieval, or conversation redesign.

Q: Why do agent loops create token problems that normal dashboards miss?

A: Because the expensive behaviour often happens inside a nested span, not at the top level. A retrying tool call, oversized retrieval, or sub-agent loop can consume most of the budget while the parent trace still looks ordinary. Span-level token attribution is what reveals the real cost driver.

Technical breakdown

Prompt and completion tokens expose different failure modes

Prompt tokens measure what the model received, while completion tokens measure what it generated. Those two counts separate input growth from output growth, which is essential when a cost spike could come from a longer system prompt, a larger retrieval payload, a missing stop sequence, or an unconstrained answer. For reasoning models, internal reasoning tokens can also affect billing and latency even when the visible response looks short. At production scale, this split is more useful than a single total because it maps directly to different debugging paths.

Practical implication: Track prompt and completion tokens separately so teams can identify whether to trim inputs, cap outputs, or inspect reasoning-heavy calls.

Context window utilisation is a percentage, not a guess

Context-window utilisation is the share of the model’s maximum input capacity consumed by a request. Measuring it as a percentage matters because a request can look acceptable in absolute token count while still sitting uncomfortably close to the limit once tenant history, retrieved chunks, and tool descriptions are included. That is why request-level visibility by route, tenant, and model is more useful than median usage. When utilisation climbs, the most effective fixes are usually truncation, summarisation, retrieval narrowing, or smaller intermediate context between agent steps.

Practical implication: Alert on p95 utilisation by route or tenant so teams can correct overflow risk before users hit failures.

Span-level attribution finds runaway agent loops

Agent traces often contain multiple nested steps, and the expensive one is rarely the top-level call. A retrying tool invocation, a retrieval step that drags in oversized chunks, or a sub-agent loop can dominate token consumption while the parent trace still appears normal. Span-level token attribution solves that by attaching usage to each step in the trace tree, then rolling it up to the parent. That gives teams a way to sort by cost, latency, or output volume and see which step is consuming the budget.

Practical implication: Instrument every agent step so teams can isolate expensive retries, tool calls, and retrieval spans instead of guessing from aggregate totals.

NHI Mgmt Group analysis

Token observability is becoming a governance control, not just a FinOps metric. The article shows why a single daily or monthly total is too coarse for production AI systems. When the same spike can be caused by prompt bloat, context pressure, or an agent loop, leaders need telemetry that supports accountability at the call and span level. For AI governance teams, the practical conclusion is that token data belongs in the control plane, not only in billing dashboards.

Per-step visibility is the right abstraction for agentic AI risk. Agent workflows fail in places that conventional request logging cannot isolate, especially when retries, retrieval, and sub-agents amplify consumption inside a trace. That creates a distinct control problem for AI systems that use delegated tools and service credentials, because the system can behave within policy at the top level while an inner step burns through budget or context. The practitioner implication is to govern the path, not just the outcome.

Context-window pressure is a named governance gap in production LLMs. The article makes clear that proximity to model limits is operationally invisible unless it is measured as a percentage and tracked by route or tenant. That matters because context overflow is often the first visible sign that prompt design, retrieval sizing, or conversation history management is out of control. Practitioners should treat context utilisation as an early warning metric for brittle AI workflows.

LLM telemetry should be linked to the identity of the calling system. In environments where AI features use delegated tools, API credentials, or workload identities, token growth can signal misuse, drift, or uncontrolled automation. This is where NHIs intersect with AI governance: the model is not acting alone, it is acting through credentials and runtime permissions. The practical conclusion is to connect token traces to workload identity and access boundaries.

Braintrust’s framing reflects where the market is heading: from aggregated monitoring to accountable AI operations. The interesting shift is not the tool choice, but the operational model. Teams are moving toward prompt versioning, trace-level attribution, and stepwise evaluation because they need evidence for debugging, cost control, and governance at the same time. The practitioner takeaway is to build telemetry that can support both engineering triage and policy review.

What this signals

Token telemetry should be treated as an operational signal for AI governance teams, especially where models sit inside agentic workflows that consume credentials and external context. Once trace data is tied to workload identity and prompt versions, teams can distinguish normal growth from unsafe drift and enforce boundaries with more confidence. The most useful next step is to align token visibility with NIST AI Risk Management Framework controls and the OWASP Agentic AI Top 10 where agent behaviour is in scope.

Context-window pressure is a useful named concept for practitioners because it captures a failure mode that often appears before obvious outages. Requests that sit too close to model limits tend to create brittle retries, retrieval overreach, and uneven latency, which makes them hard to diagnose from aggregate metrics alone. Teams should watch for routes whose p95 utilisation keeps climbing, then redesign prompts, retrieval size, or intermediate context transfer before the failure becomes routine.

For practitioners

• Instrument every LLM call with usage telemetry Capture prompt tokens, completion tokens, cached-token fields, and reasoning-token fields on every call so production issues can be traced to the exact request rather than inferred from monthly totals.
• Track context-window utilisation by route and tenant Convert prompt_tokens into a percentage of model capacity, then alert on high p95 utilisation for specific tenants, features, or routes before overflow starts disrupting service.
• Break agent traces into inspectable spans Attach token counts, latency, retry count, and tool metadata to each span so oversized retrievals or runaway loops can be isolated without reading the full parent trace.
• Separate prompt growth from output growth in dashboards Build views that distinguish larger inputs from longer completions so prompt drift, missing stop conditions, and retrieval bloat do not get collapsed into one generic cost spike.
• Tie AI telemetry to workload identity Link trace data to the service account, API key, or workload identity used by the AI system so abnormal usage can be investigated as an access and governance issue.

Key takeaways

• Aggregate token totals hide the real failure mode, because prompt bloat, context pressure, and agent loops need different control responses.
• Per-call and per-span telemetry turns token tracking into an operational control that supports debugging, cost management, and AI governance.
• The practical shift is from monitoring volume to attributing behaviour, especially where AI systems run through delegated tools and workload identities.

Key terms

• Prompt Tokens: Prompt tokens are the input units a model processes before generating a response. They include system prompts, user prompts, retrieved context, tool schemas, and conversation history, which means rising prompt tokens usually point to input growth rather than output expansion.
• Completion Tokens: Completion tokens are the units generated by the model in its response. They help distinguish long answers, missing stop conditions, and reasoning-heavy outputs from simple input bloat, and they are essential for understanding latency and cost behaviour in production traces.
• Context Window: The context window is the text a model receives at one time, including prompts, retrieved documents, and conversation history. Security teams care about it because it becomes the practical boundary between trusted instructions and untrusted content, especially when the application assembles that text automatically.
• Token-Level Attribution: The practice of binding each action token to the originating identity and authorisation context. It allows investigators and auditors to trace what an AI agent did, for whom, and under what consent, even when the workflow spans multiple systems and tool calls.

What's in the full article

Braintrust's full guide covers the operational detail this post intentionally leaves for the source:

• Exact SDK field names for OpenAI and Anthropic usage payloads, including cached-token and reasoning-token fields
• OpenTelemetry GenAI attribute examples for teams standardising on portable observability conventions
• Auto-instrumentation setup patterns for Python and TypeScript, including supported frameworks and streaming responses
• Dashboard and trace workflow details for rolling up token usage by prompt version, tenant, feature, and route

👉 The full Braintrust guide includes implementation details for SDK payloads, tracing, and OpenTelemetry conventions.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management in production environments. It helps practitioners connect identity controls to the operational realities of AI and automation.