How should teams track LLM token usage when prompts, retrieval, and agent steps all contribute?

Track token usage at the right measurement grain

Token accounting only becomes useful when it distinguishes the different places cost and latency accumulate. A single total per user or per day hides whether the issue comes from longer prompts, larger retrieved context, or an agent looping through repeated steps. Treat usage as a structured trace attribute, not just a billing line item.

The practical split is per call, per request, and per span. Per call tells you what one model invocation consumed, per request shows the full end-to-end cost of a user action, and per span shows where the work happened inside the trace. That separation makes the production problem visible instead of collapsing it into one number.

For teams using retrieval, attach the prompt token count, completion token count, and any retrieval-fed context size to the same trace record. If retrieval growth is the driver, you should see the prompt side rise first. If an agent is the driver, the repeated span pattern will show multiple small calls instead of one unusually large call.

Why prompt, retrieval, and agent steps need separate accounting

Prompt growth, retrieved context, and agent orchestration fail in different ways, so they need different counters. Prompt growth usually means the request is carrying too much conversation history or too much unfiltered context. Retrieval growth often means the system is over-inserting documents, pulling in duplicate passages, or failing to rank by relevance. Agent-step growth usually means the workflow is looping, re-planning, or re-calling tools more than expected.

When those sources are blended together, teams often optimise the wrong layer. A model may appear expensive when the real issue is repeated retrieval, or a retrieval system may appear noisy when the dominant cost is an agent that keeps re-asking the same question. Separate accounting lets engineering teams move from guesswork to targeted fixes.

That distinction also matters for debugging regressions. If per-request usage climbs but per-span usage is flat, the likely change is in request composition. If one span type suddenly dominates, the issue is probably in orchestration, tool calling, or loop control. If prompt tokens climb while completion tokens stay stable, the problem is usually upstream context inflation rather than model verbosity.

Make traces answer the production question

Token metrics should be attached to traces in a way that answers the operational question immediately: is the problem input growth, output growth, or repeated agent work? A trace that carries only a total token count cannot answer that question. A trace with span-level labels can show which step expanded, which retrieval stage contributed, and whether the model call itself changed.

This is especially useful for applications that combine retrieval-augmented generation, tool use, and multi-step agents. In those systems, the model is not the only consumer of tokens. The surrounding orchestration can multiply usage even when the base prompt appears stable. Observability should therefore follow the request path, not just the model endpoint.

For a useful implementation, teams usually standardise a small set of fields: request ID, span ID, prompt tokens, completion tokens, retrieved-context tokens if available, and a step label for retrieval or agent action. Those fields let product, platform, and FinOps teams discuss the same event without arguing over a blended total.

Risk and Threat Considerations

Without granular token tracking, teams can miss runaway consumption, hidden retry loops, and prompt or retrieval inflation until cost or latency becomes visible to users. In agentic and retrieval-heavy systems, a small logic error can create repeated model calls that look normal in aggregate but are expensive in practice.

Failure mechanism: A total-only counter hides whether usage is driven by larger inputs, larger outputs, or repeated intermediate steps, so the team cannot see the actual production fault. That makes it harder to spot loops, unnecessary context expansion, or orchestration mistakes before they become a sustained cost or performance issue.

Impact: Teams lose the ability to attribute spend, diagnose regressions, and decide whether the fix belongs in prompt design, retrieval filtering, or agent control flow. The result is slower incident response, less reliable budgeting, and repeated tuning in the wrong place.

Practitioner Guidance

What to prioritize: Start with trace-level instrumentation that preserves per-call and per-span token fields before you spend time on dashboards. If you cannot separate prompt and completion tokens in the trace, you will struggle to explain usage changes later.

What to measure: Watch the ratio between prompt tokens, completion tokens, retrieved-context size, and the number of agent steps per request. A rise in any one of those ratios usually tells you where the inefficiency sits, and it is more actionable than a single aggregate total.

Common mistake: Treating total tokens as the primary metric for everything. That may be adequate for billing, but it is too coarse for diagnosing production behaviour in systems where retrieval and orchestration contribute materially.

Practitioner takeaway: The useful unit of analysis is not “how many tokens did this user use,” but “which stage created the tokens, and did it do so once or in a loop?”

Related resources from NHI Mgmt Group

• How should security teams design agent workflows to avoid unnecessary user prompts?
• How should security teams handle trust assumptions in LLM and AI agent workflows?
• How should security teams enforce LLM compliance across prompts and retrievals?
• How should security teams govern LLM and agent access in production?