Braintrust shows how span-level tracing cuts LLM costs

At a glance

What this is: Braintrust argues that LLM cost control starts with span-level tracing, because only per-call visibility reveals where tokens and spend accumulate across prompts, tools, retries, and model choices.

Why it matters: For IAM, NHI, and broader security teams building AI workflows, the same tracing discipline helps govern agent behaviour, tool use, and quality gates before production changes expand risk or cost.

By the numbers:

• Braintrust says AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers.
• Braintrust says 28% of secrets incidents now originate outside code repositories, in Slack, Jira, and Confluence, and are 13% more likely to be categorised as critical than code-based leaks.
• Braintrust says 64% of valid secrets leaked in 2022 are still valid and exploitable today, showing that detection alone does not resolve exposure.

👉 Read Braintrust's guide to reducing LLM costs with tracing and evals

Context

LLM cost governance fails when teams can only see a total bill, not the prompts, tool calls, retries, and model decisions that produced it. In production, token spend usually grows through context bloat, agent loops, and model overuse rather than one obvious mistake, which makes simple dashboard reporting too coarse for control. For teams building AI-enabled products, this is also an identity and access issue because tools, models, and agents act through credentials and runtime permissions.

Braintrust's article frames cost control as an observability problem first and a model-selection problem second. That is the right order for most production AI programmes: if a team cannot attribute spend to a span, it cannot confidently optimise prompts, route tasks to cheaper models, or prove that changes preserve output quality. The pattern is typical across agentic and RAG-heavy systems, not an edge case.

The operational lesson extends beyond budget management. In AI systems, cost spikes often correlate with uncontrolled context growth, excessive tool use, and poorly governed retry behaviour. Those same mechanisms can increase the blast radius of misuse, especially when agents rely on secrets or delegated access to act across internal tools.

Key questions

Q: How should teams reduce LLM costs without breaking production quality?

A: Start with request-level tracing so you know which prompts, models, retrieval steps, or tool calls drive cost. Then test cheaper alternatives on real production traces and require evaluation gates before release. That combination reduces spend while preserving quality, rather than simply shifting cost into retries or manual review.

Q: Why do agentic AI systems often cost more than teams expect?

A: Because the visible user request usually hides many internal steps. Agents may call tools repeatedly, retrieve too much context, or trigger several model calls inside one workflow. Those extra spans add up quickly, so aggregate dashboards miss the real drivers of spend and make optimisation feel uncertain.

Q: What signals show that an LLM workflow is becoming inefficient?

A: Look for growing system prompts, retrieval steps that return oversized context, repeated tool calls, and traces where one child span is far more expensive than its siblings. Those patterns usually indicate prompt bloat, retry loops, or overpowered model selection rather than a single isolated failure.

Q: How do organisations govern cost optimisation in production AI safely?

A: Use CI/CD evals as the approval gate for prompt and model changes, and make expensive production traces reusable test cases. That gives the team a controlled way to improve spend while preserving quality and creates a durable record of known failure patterns.

Technical breakdown

Why span-level tracing is the control plane for LLM cost governance

A trace records the full execution path of a single request, including model calls, retrieval steps, tool use, and retries. When cost and token counts are attached to each span, teams can isolate the exact branch that drives spend instead of treating the request as one undifferentiated unit. That matters because one agentic request can fan out into many LLM calls, and the most expensive step is often hidden several layers deep in the trace tree. Cost governance becomes practical only when the system preserves that per-span evidence.

Practical implication: Use trace-level cost attribution to identify the specific span, prompt, or tool call that needs redesign before changing models globally.

How prompt bloat and tool loops inflate token spend

Prompt bloat happens when system instructions, examples, or retrieved context accumulate over time until every request carries unnecessary tokens. Tool loops add another layer when agents repeat failed calls, re-query the same data, or summarize content they do not need. Both patterns are easy to miss in aggregate dashboards because the waste is distributed across many spans. Timeline views and child-span inspection reveal these inefficiencies by making oversized blocks and repeated calls stand out visually, which is often enough to diagnose whether the issue is context growth, retrieval scope, or retry logic.

Practical implication: Audit traces for oversized prompts, repeated tool calls, and retrieval steps that pull more context than the task requires.

Why evals and quality gates matter in cost reduction

Lowering spend is risky if the change silently reduces answer quality, breaks tool-use correctness, or weakens refusal behaviour. Evals address that gap by scoring candidate prompts or models against a fixed dataset before release. In production AI programmes, this is the equivalent of a controlled change window: the team can test a cheaper model, tighten a prompt, or alter retrieval logic while checking whether quality stays within threshold. Tying evals into CI/CD turns cost optimisation into a governed release activity rather than an ad hoc tuning exercise.

Practical implication: Block cost-saving changes unless they clear automated eval thresholds for the behaviours that matter to your use case.

NHI Mgmt Group analysis

Span-level observability is now a governance requirement for production AI. Total-cost dashboards can tell teams that spending is increasing, but they cannot identify the prompt, tool call, or model decision causing the rise. That gap is especially problematic in agentic workflows where one request can cascade through many steps. The practical conclusion is that AI cost control belongs alongside runtime governance, not beside it.

The named concept here is token opacity debt. This is the accumulated inability to explain where tokens are consumed across prompts, retrieval, tools, and retries. Once that debt grows, organisations begin optimising blindly, often by switching models before they understand the actual cost driver. The result is a recurring governance failure: lower spend in one place and hidden waste elsewhere.

Cost reduction in AI systems cannot be separated from identity and access governance. Agents, tools, and model calls execute through credentials, service identities, and delegated permissions, so inefficient workflows can also enlarge the access surface. That intersection matters for NHIs because the same runtime path that burns tokens may also invoke privileged tools. Practitioners should treat cost tracing and credential governance as linked controls, not separate workstreams.

Quality gates are the control that keeps optimisation from becoming regression. Shorter prompts and cheaper models often look attractive until they break edge cases in production. Evals in CI/CD provide the evidence base needed to approve change, making cost savings auditable rather than anecdotal. Teams should view that discipline as part of secure AI change management, not just model evaluation.

The market signal is moving toward lifecycle governance for AI systems, not one-off tuning. Production traces become reusable test cases, which means the organisation builds a memory of past failures and cost issues into the release process. That is where AI governance is heading: from reactive dashboards to continuous control. Practitioners should design for repeatable evidence, not just one-time optimisation.

What this signals

AI cost control and AI security are converging on the same operational reality: if teams cannot see which span consumed tokens, they usually cannot see which span invoked privilege either. That is why trace telemetry should be reviewed alongside OWASP Agentic AI Top 10 controls and, where model and pipeline governance is in scope, the NIST AI Risk Management Framework. The programme signal is clear: cost transparency is becoming an access-control adjacent concern, not just a FinOps concern.

Token opacity debt: the more a team lets prompts, retrieval, and tool loops grow without measurement, the harder it becomes to explain both cost and behaviour. In practice, that debt slows incident review, weakens change approval, and makes model routing decisions harder to defend. Practitioners should expect cost tracing to become part of standard AI control evidence, much like logging and eval history.

Production traces will increasingly serve as the evidence layer for AI governance programmes. The teams that can turn expensive requests into durable eval cases will improve cost discipline faster than those relying on periodic dashboard reviews. That creates a stronger feedback loop for engineering, security, and product owners who need to defend both spend and output quality.

For practitioners

• Implement span-level cost attribution Attach estimated cost and token counts to every request span so product, engineering, and security teams can identify the exact prompt, retrieval step, or tool call driving spend.
• Inspect agent loops and retrieval scope Review trace trees for repeated tool calls, oversized context windows, and retrieval steps that pull more data than the task requires, then narrow the workflow before switching models.
• Run prompt and model experiments on production traces Compare candidate prompts and models against the same real inputs, using quality scores to verify that a cheaper option still meets output standards.
• Block cost-cutting changes with CI/CD eval gates Require automated evals for every prompt or model change so merge approval depends on preserved quality, not only reduced token usage.
• Convert expensive traces into permanent test cases Promote costly or broken production traces into reusable eval cases so future changes are checked against known failure patterns before release.

Key takeaways

• LLM cost overruns usually come from hidden span-level waste, not from one obvious model choice.
• Production traces, prompt experiments, and eval gates turn cost reduction into an auditable control process.
• AI programmes that ignore token visibility also risk missing the linked identity and access behaviour inside agent workflows.

Key terms

• Span-Level Tracing: Span-level tracing records each step in an LLM request as an individual unit, including model calls, retrieval, tool use, and retries. It lets teams attribute cost and behaviour to the exact action that produced it, which is essential when one request fans out into many hidden subcalls.
• Token Opacity Debt: Token opacity debt is the accumulation of cost and usage that cannot be explained at the prompt, tool, or model level. As the debt grows, teams lose the ability to optimise safely, because they can no longer tell whether savings came from a real efficiency gain or from broken behaviour.
• Activation Gate: An activation gate is the review point that must be satisfied before an agent or workload is allowed to operate in production. It is an identity control, not a product feature, because it decides whether the runtime is permitted to act at all.
• Context bloat: The accumulation of tool definitions, schemas, and metadata in an agent’s working context until reasoning capacity is consumed by exposure rather than task execution. It matters because too many connected tools can reduce performance, obscure intent, and make access governance harder to reason about.

What's in the full article

Braintrust's full article covers the operational detail this post intentionally leaves for the source:

• Side-by-side examples of trace views, timeline views, and child-span drill-downs for identifying high-cost LLM steps
• Concrete prompt editing patterns that reduce token usage without changing the business logic of the workflow
• Model comparison examples showing cost and quality scores across different model classes
• CI/CD eval workflow details, including how to block merges when quality thresholds are not met

👉 Braintrust's full article covers the trace views, prompt experiments, and eval workflow in operational detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to connect identity controls to emerging AI and automation risk.