Topic starter
20/08/2026 6:00 pm
TL;DR: Aggregate token totals hide why production LLM costs and latency spike, according to Braintrust. Token telemetry only governs well when it links per-call, request-level, and span-level data to traces and agent steps.
Editorial analysis by NHI Mgmt Group, based on content published by Braintrust: “How to track LLM token usage (2026): Prompt, completion, context window, and per-step visibility”.
Key questions
Q: Why do token totals fail to explain production latency and cost spikes?
A: Because the same spike can come from very different causes.
Q: What are the signs that LLM token tracking is failing as an operational control?
A: The main signs are unexplained cost spikes, rising p95 latency without a visible prompt change, and trace views that cannot identify the expensive step.
Q: How should teams track LLM token usage when prompts, retrieval, and agent steps all contribute?
A: Track usage at three levels: per call, per request, and per span.
Practitioner guidance
- Implement token telemetry at the trace boundary Attach prompt, completion, and reasoning-token usage to every LLM call so the operational record explains the request path, not just the monthly total.
- Separate prompt and completion metrics Report input and output tokens independently, then compare them across prompt versions, models, and routes to identify whether input growth or output growth is driving the spike.
- Set context utilization thresholds Calculate request-level context-window utilisation as a percentage and alert on p95 pressure by tenant, feature, model, and route before overflow errors appear.
Bottom line: Aggregate token totals are too coarse to explain why production LLM systems slow down or become expensive.
Glossary terms: Context Window, Token Telemetry, Span-Level Attribution, Answer Drift
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
This topic was modified 2 days ago by NHI Mgmt Group
20/08/2026 6:38 pm
Token telemetry is becoming a governance control, not a reporting convenience. Once LLM systems are embedded in production workflows, aggregate usage totals are too blunt to support decision-making. Teams need token data attached to traces, prompts, and spans so they can explain behaviour, not just count it. The practical implication is that observability and governance have to be designed together.
A few things that frame the scale:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
- Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
A question worth separating out:
Q: What is the difference between prompt-token tracking and span-level token tracking?
A: Prompt-token tracking tells you how much input a call consumed, while span-level tracking shows which step in the workflow burned the tokens. The first is useful for prompt design and context sizing. The second is essential for agent traces, where retries, tool calls, and sub-agent loops can dominate cost.
👉 Read our full editorial: LLM token tracking reveals where production behavior breaks down