Topic starter
20/08/2026 6:00 pm
TL;DR: Aggregate token totals hide the real causes of LLM cost and latency spikes, and teams need per-call, context-window, and per-span visibility to distinguish prompt bloat, context pressure, and runaway agent loops, according to Braintrust’s 2026 guide. The operational lesson is that token telemetry becomes a governance control only when it is attached to traces, prompts, and agent steps in production.
Editorial analysis by NHI Mgmt Group, based on content published by Braintrust: “How to track LLM token usage (2026): Prompt, completion, context window, and per-step visibility”.
Key questions
Q: How should teams track LLM token usage when prompts, retrieval, and agent steps all contribute?
A: Track usage at three levels: per call, per request, and per span.
Q: Why do token totals fail to explain production latency and cost spikes?
A: Because the same spike can come from very different causes.
Q: What are the signs that LLM token tracking is failing as an operational control?
A: The main signs are unexplained cost spikes, rising p95 latency without a visible prompt change, and trace views that cannot identify the expensive step.
Practitioner guidance
- Implement token telemetry at the trace boundary Attach prompt, completion, and reasoning-token usage to every LLM call so the operational record explains the request path, not just the monthly total.
- Separate prompt and completion metrics Report input and output tokens independently, then compare them across prompt versions, models, and routes to identify whether input growth or output growth is driving the spike.
- Set context utilization thresholds Calculate request-level context-window utilisation as a percentage and alert on p95 pressure by tenant, feature, model, and route before overflow errors appear.
Bottom line: Aggregate token totals are too coarse to explain why production LLM systems slow down or become expensive.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
20/08/2026 6:38 pm
Token telemetry is becoming a governance control, not a reporting convenience. Once LLM systems are embedded in production workflows, aggregate usage totals are too blunt to support decision-making. Teams need token data attached to traces, prompts, and spans so they can explain behaviour, not just count it. The practical implication is that observability and governance have to be designed together.
A few things that frame the scale:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
- Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
A question worth separating out:
Q: What is the difference between prompt-token tracking and span-level token tracking?
A: Prompt-token tracking tells you how much input a call consumed, while span-level tracking shows which step in the workflow burned the tokens. The first is useful for prompt design and context sizing. The second is essential for agent traces, where retries, tool calls, and sub-agent loops can dominate cost.
👉 Read our full editorial: LLM token tracking reveals where production behavior breaks down