AI observability lowers LLM spend by exposing costly workflow spans

At a glance

What this is: This guide argues that LLM cost problems are usually workflow problems, because the expensive steps sit inside traces and spans rather than at the headline model-call level.

Why it matters: It matters to AI and platform teams because cost control, quality control and release governance all depend on seeing where prompts, retries, retrieval and tool calls actually inflate spend.

By the numbers:

• A request that looks like a $0.02 model call can become a $0.38 agent workflow once long context, extra calls, tool calls and retries are counted.

Context

LLM cost management is often a governance problem disguised as a billing problem. Aggregate spend can rise for reasons that remain invisible at the model or application level, especially when retrieval, retries, tool calls and agent steps are distributed across a workflow.

In practical terms, trace-level observability matters because it ties cost to the actual execution path, including prompts, spans and tool calls. That gives AI teams a control point for spend reduction without treating lower cost and lower quality as the same outcome.

Key questions

Q: Why do aggregate dashboards miss the real source of LLM cost growth?

A: Because dashboards aggregate spend after the workflow has already run, they hide the span, prompt or retry pattern that created the increase. LLM cost often comes from the shape of execution, not the headline model price, so teams need trace-level attribution to see which step is multiplying token usage.

Q: When should teams prioritise trace-level observability over billing reports?

A: As soon as production workflows include retrieval, retries, tool calls or agents. Billing reports tell you that cost changed, but trace-level observability tells you whether the driver is context bloat, repeated execution or an expensive workflow step that can be redesigned.

Q: What do security and AI teams get wrong about lowering LLM spend?

A: They often optimise cost in isolation and skip evaluation, which can make a cheaper prompt or model look successful while degrading quality. The safer approach is to validate every cost-saving change against representative cases before it reaches production.

Q: How should teams separate cost monitoring from cost governance in AI workflows?

A: Cost monitoring answers what the bill was, while cost governance answers which workflow step is allowed to generate that bill. In practice, that means tracing spans, limiting unnecessary loops, and tying prompt or model changes to release controls and quality thresholds.

Technical breakdown

Why aggregate dashboards miss LLM cost drivers

A dashboard can tell you that usage went up, but it cannot explain which span caused the increase. In LLM systems, the expensive unit is often the workflow, not the request. A single user action may trigger retrieval, reranking, generation, validation and retries, each with different token counts and model costs. Without span-level instrumentation, a team sees the final bill but loses the causal chain. That makes it hard to distinguish context bloat from agent loops or a broken tool response from normal traffic growth. Practical implication: teams need cost attribution at the span level, not only application-level billing summaries.

Practical implication: Instrument traces so spend can be traced to the exact prompt, retrieval step or retry that created it.

How tool calls and retries inflate token spend

Tool calls and retries create hidden multipliers because each failed or intermediate step often resends the full prompt and surrounding context. Agentic workflows are especially vulnerable because they can loop through multiple model calls before producing a result, and every loop may carry the same long context window. The result is that the real cost of a request is not the first model response, but the total across nested spans. This is why cost reduction usually starts with fixing the workflow shape, not only changing the model. Practical implication: identify repeated spans, malformed outputs and oversized context blocks before assuming the model itself is the problem.

Practical implication: Look for repeated tool spans, schema failures and looped calls that force the model to reprocess the same context.

Why eval-backed release control is part of cost governance

Cost optimisation without evaluation is just cheaper guessing. A shorter prompt may reduce tokens but still degrade accuracy, while a smaller model may work for extraction or routing but fail on reasoning-heavy tasks. Evals turn cost changes into measurable release decisions by checking whether a lower-cost variant still meets the quality bar on representative cases. That matters because LLM spend and model risk are coupled: a cost reduction that breaks output quality can create downstream operational cost. Practical implication: tie prompt and model changes to evaluation gates before they reach production.

Practical implication: Use evaluations as the release gate for prompt or model changes that are intended to reduce spend.

Threat narrative

Attacker objective: The operational outcome is not compromise but uncontrolled spend, where the real cost driver stays hidden until the bill arrives.

1. Entry begins when a production request triggers a workflow that fans out into retrieval, generation, tool use and retry logic, multiplying the number of model calls inside one trace.
2. Escalation occurs as larger context windows, repeated prompt resends and agent loops compound the token load across nested spans, turning a low-cost request into an expensive workflow.
3. Impact is sustained overspend and weak root-cause visibility, because aggregate billing hides the exact span that created the cost spike.

Breaches seen in the wild

• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Trace-level visibility has become the cost-control layer for AI systems. Aggregate monitoring can tell teams that spend is rising, but it cannot show which span, prompt or tool call created the increase. That makes optimisation guesswork until the workflow is broken down into instrumented steps. For practitioners, the governance question is no longer whether to monitor AI, but whether cost attribution exists at the point where the spend is generated.

The real cost problem is workflow amplification, not model pricing. The article’s examples show that retrieval, retries, agent loops and context growth can turn a seemingly cheap request into a materially expensive trace. That is a control problem because the same request shape can recur hundreds or thousands of times. Practitioners should treat repeated spans, long context windows and looped tool execution as cost anomalies, not normal variation.

Cost observability and quality governance now need to move together. Lowering spend without eval-backed release control simply shifts risk from finance to product reliability. The pattern here is useful beyond LLM cost optimisation: any AI governance programme that cannot tie change control to measured output quality will struggle to prove that savings are safe.

Context bloat is a distinct governance failure mode. Redundant instructions, stale few-shot examples and unfiltered retrieval output accumulate quietly over time, creating a hidden tax on every request. That is why AI teams need to manage prompt lifecycle with the same discipline they apply to code or policy changes. Practitioners should review prompts as living production assets, not static configuration.

AI cost governance is becoming an identity and control issue as well as a tooling issue. Once workflows include agents, tools and delegated actions, the real question is who or what is authorised to create expensive execution paths. That is where AI observability intersects with IAM thinking: control the action path, not just the interface. Practitioners should treat high-cost execution patterns as a sign that authorisation and release governance are not aligned.

From our research library:

• 96% of security operations teams report critical blind spots, most commonly in cloud infrastructure (74%) and identity and access behaviour (67%).
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Workflow amplification is now the main LLM cost risk. Once retrieval, retries and agent loops are part of the execution path, a low-cost call can become an expensive trace. Teams should assume the cost driver sits inside the span tree until proven otherwise.

Observability must become a release control, not just a reporting layer. If AI teams cannot connect cost changes to eval-backed validation, they will continue to trade quality for savings without noticing until production users do.

The broader signal for practitioners is that AI governance is shifting toward execution-path control. As agents and delegated tools become more common, teams need to know which spans are generating cost, which spans are generating risk, and whether the same control gate can govern both.

For practitioners

• Map cost to trace spans Instrument every model call, retrieval step, tool execution and retry so cost is visible at the span level rather than only in monthly totals.
• Flag workflow amplification patterns Review traces for repeated loops, oversized context windows, schema failures and silent retries that multiply token usage across a single user request.
• Optimise prompts before changing models Test shorter prompts, cleaner instructions and tighter retrieved context against the same inputs to see whether token reduction preserves quality.
• Use evals as the release gate Convert representative traces into evaluation cases and block production rollout when a lower-cost prompt or model does not meet the quality threshold.
• Separate cheap steps from reasoning steps Route extraction, classification and formatting to lower-cost models where the evaluation suite shows they meet the bar, and reserve frontier models for the steps that need them.

Key takeaways

• LLM cost inflation usually comes from workflow design, not the base model price, because retries, tool calls, context growth and agent loops compound inside production traces.
• Trace-level observability gives teams the causal detail that billing dashboards cannot, making it possible to identify the exact span or prompt that is driving overspend.
• The safest path to lower cost is to pair prompt and model optimisation with eval-backed release control so savings do not quietly degrade output quality.

Key terms

• Trace Observability: Trace observability is the ability to inspect a workflow step by step, including inputs, decisions, retries, and outputs. For AI agents, traces show where a loop is wasting tokens, hitting permission barriers, or failing to converge. That visibility supports debugging, cost control, and post-run review.
• Span: A span is one discrete operation inside a trace, such as a retrieval query, prompt assembly step, or model call. Spans make it possible to pinpoint which part of an AI workflow consumed time, introduced error, or passed bad context to the next stage.
• Context bloat: The accumulation of tool definitions, schemas, and metadata in an agent’s working context until reasoning capacity is consumed by exposure rather than task execution. It matters because too many connected tools can reduce performance, obscure intent, and make access governance harder to reason about.
• Eval-Backed Release Control: Eval-backed release control uses repeatable test cases and scoring to verify that a prompt, model, or workflow change is safe before it reaches production. It turns optimisation into a governed change process, so lower cost does not come at the expense of output quality or task reliability.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security and secrets management for practitioners building stronger control frameworks. It helps security leaders connect identity governance to the broader operational decisions their programmes depend on.