The main signs are unexplained cost spikes, rising p95 latency without a visible prompt change, and trace views that cannot identify the expensive step. If teams cannot separate prompt, completion, and span-level usage, the telemetry is too coarse to support investigation or governance.
Why token tracking fails before the model itself looks “wrong”
operational control failures are usually visible first in the telemetry, not the model output. When usage accounting is too coarse, teams lose the ability to explain where spend, latency, and load are coming from, which turns token tracking from a control into an after-the-fact estimate. The failure is often gradual: one more proxy, cache, gateway, or framework layer is enough to blur the trail.
That is why the strongest warning sign is not simply high usage. It is unbounded consumption patterns that cannot be tied back to a prompt, tenant, user, or workflow step. Once you cannot tell whether tokens were spent on input, output, retries, or hidden orchestration, the control has stopped being operationally useful.
In practice, this usually means the organisation has a measurement problem, not a model problem. Token tracking has to survive batching, retries, tool calls, guardrails, and middleware if it is going to support incident review, cost attribution, or usage governance.
Where coarse telemetry shows up first
The most obvious signal is cost drift that does not match product behaviour. If spend jumps but prompt volume, user traffic, and release activity look stable, the accounting layer is probably hiding duplicate calls, long completions, or background agent activity. A second signal is latency inflation that appears without a visible prompt change, especially when p95 rises while average input size stays flat.
Another common symptom is the inability to isolate a single expensive step in traces. If the team can only see a whole request, but not prompt, completion, tool invocation, and span-level usage separately, then the tracing model is too blunt for debugging. That limitation matters because it prevents teams from distinguishing inefficient prompting from retry storms, routing overhead, or downstream tool amplification.
When those views are missing, the control can still report totals, but it cannot explain variance. For governance, that is the difference between “we know usage increased” and “we know which action consumed the budget.”
What the control must prove, not just report
A working token control should let operators answer three questions quickly: what was requested, what the model returned, and what intermediate step consumed the most tokens. If the answer requires manual log reconstruction or guesswork, the system is under-instrumented.
This is where identity-focused evaluation criteria and PoC tests become useful, because the right test is not “does the tool show a dashboard?” but “can it separate usage by prompt, completion, and internal span with enough fidelity to support investigation?” The control should also preserve enough detail to spot abnormal bursts, repeated retries, and over-consumption by a single workflow.
Good token tracking is therefore less about raw counts and more about attribution quality. If the data cannot be sliced by tenant, workflow, request type, or step, it will not support cost governance or root-cause analysis when something goes wrong.
Risk and Threat Considerations
Coarse token telemetry creates both operational risk and abuse risk. Internally, it hides runaway prompts, inefficient chains, and retry loops; externally, it makes abuse harder to spot when an attacker or misconfigured automation drives unusually large model consumption.
Failure mechanism: Usage is aggregated too early or at too high a level, so the organisation loses the ability to attribute cost and latency to the exact prompt, completion, or span that caused them. That weakens anomaly detection and makes it harder to identify whether the issue is a benign workload change, a defect, or active misuse.
Impact: Teams lose trust in the metric, spend more time reconstructing incidents, and may miss a sustained cost or availability problem until it has already affected users or budget.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10 addresses the attack and risk surface, while NIST SP 800-53 Rev 5, CIS Controls v8 and NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Non-Human Identity Top 10 | NHI-02 — Secret Leakage | Coarse token telemetry often hides the same usage paths that reveal leaked secrets or runaway consumption. |
| NHI-07 — Long-Lived Secrets | Operational blind spots often appear when long-lived credentials drive repeated, hard-to-attribute model usage. | |
| NHI-05 — Overprivileged NHI | Excessive access can amplify hidden token consumption and obscure which workflow caused the cost. | |
| Recommendation — Instrument token and trace data well enough to expose secret-driven abuse and investigate anomalous spend. Track token usage by actor and workflow so long-lived credentials cannot consume services unnoticed. Limit workflow and service privileges so high-volume token use stays attributable and bounded. | ||
| NIST SP 800-53 Rev 5 | AU-6 — Audit Record Review, Analysis, and Reporting | Token tracking failures are observability failures that weaken review and anomaly analysis. |
| AU-12 — Audit Record Generation | The answer depends on generating sufficiently detailed records for prompt, completion, and span usage. | |
| AU-2 — Event Logging | Trace views and coarse telemetry are logging design issues that determine whether investigations are possible. | |
| Recommendation — Review usage records for abnormal spikes and unexplained latency before accepting the control as effective. Generate granular usage records that capture the fields needed to separate prompt, completion, and span costs. Log the model events and sub-steps needed to reconstruct expensive requests without manual correlation. | ||
| CIS Controls v8 | CIS-8 — Audit Log Management | Effective token tracking depends on logs that support investigation, correlation, and anomaly review. |
| CIS-14 — Security Awareness and Skills Training | Teams often misread coarse token metrics, so operators need to recognise what a healthy trace should show. | |
| Recommendation — Retain and review model usage logs with enough detail to separate normal traffic from anomalous consumption. Train operators to distinguish model demand from telemetry gaps before escalating a usage spike. | ||
| NIST AI RMF | MAP 1 — Govern and Map | Token tracking is a measurement and governance problem that needs defined traceability and accountability. |
| MEASURE 1 — Measure AI risks and impacts | The question is about whether measurements are detailed enough to support operational control decisions. | |
| Recommendation — Define ownership, measurement boundaries, and escalation paths for model usage telemetry. Measure whether your telemetry can attribute spend, latency, and workflow load to specific model steps. | ||
Practitioner Guidance
What to verify: Confirm that telemetry can show prompt size, completion size, retries, tool calls, and span-level consumption for the same request. If any one of those is missing, you do not yet have enough evidence to explain a cost or latency shift.
What good looks like: A reviewer should be able to take one request and trace the expensive step without leaving the observability stack. If the investigation depends on correlating three or more systems by hand, the control is too weak for operational governance.
Decision rule: If a usage spike cannot be decomposed into input, output, and intermediate work, treat it as an observability defect first and a model-efficiency issue second.
Practitioner takeaway: Token tracking is only a real control when it can explain variance, not merely total it. If it cannot attribute spend and latency to a specific step, it will fail the first serious investigation.
Related resources from NHI Mgmt Group
- What are the signs that an LLM deployment is failing its access-control and leak-prevention checks?
- What are the signs that Exchange Online PowerShell access is failing because of identity or session control issues?
- What are the signs that LLM output controls are failing in production?
- What are the signs that a control environment is failing in practice?