Braintrust shows how span-level tracing cuts LLM costs

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Braintrust's blog post: Span-level tracing reveals the hidden cost drivers in LLM workflows by attaching estimated spend and token counts to each request step.

Why it matters: This matters because teams managing AI, IAM-adjacent service workflows, and governance controls need to know which prompt, model, or tool call is inflating spend before they can safely optimise it.

Context

LLM cost control is fundamentally a visibility problem: if teams can only see total spend, they cannot tell whether the waste comes from prompt bloat, retrieval overreach, retries, or model choice. That makes optimisation brittle, because organisations end up changing the visible variable instead of the true driver.

The governance issue is especially sharp where LLM systems use tool calls, retrieval, and chained model interactions. In those environments, a single request can fan out into many costly spans, so the control point is not just model selection but the traceability of every step that consumes tokens.

Key questions

Q: How should teams reduce LLM costs without breaking production quality?

A: Start with request-level tracing so you know which prompts, models, retrieval steps, or tool calls drive cost. Then test cheaper alternatives on real production traces and require evaluation gates before release. That combination reduces spend while preserving quality, rather than simply shifting cost into retries or manual review.

Q: Why do LLM costs often stay hidden until production?

A: Testing traffic is usually too small and too clean to expose context bloat, retry loops, or inefficient tool use. Production traffic introduces real user patterns, larger retrievals, and chained calls, which is why aggregate dashboards can miss the actual cost drivers until the bill rises.

Q: What breaks when teams rely only on cloud billing dashboards to manage LLM spend?

A: Cloud billing shows totals, but it does not explain which prompt, agent, model, or retry pattern drove the increase. That leaves teams unable to separate expensive design choices from normal usage. In practice, cost issues are detected late, budgets are enforced poorly, and optimization becomes reactive instead of continuous.

Q: Should teams switch to cheaper models or fix prompts first?

A: Usually fix prompts and workflow structure first, because oversized prompts, retrieval bloat, and retry loops often account for a large share of cost. Switch models when tracing shows the prompt is already tight and the remaining spend comes from a model that is unnecessarily strong for the task.

Technical breakdown

Why total-spend dashboards miss the real cost drivers

Aggregate dashboards collapse many execution steps into one number, which hides where tokens are actually consumed. In an LLM workflow, the cost may sit in a long system prompt, a retrieval step that pulls too much context, repeated tool calls, or a fallback path that fires only under failure. A trace breaks the request into spans, so each call can carry its own token count and estimated cost. That changes investigation from guesswork to attribution. Instead of assuming the model is expensive, teams can see whether the bill comes from prompt design, orchestration logic, or repeated execution paths.

Practical implication: use trace-level cost data before changing models or prompts.

How trace trees expose expensive tool calls and retry loops

Tool calls often dominate spend in agentic and retrieval-augmented systems because each tool can trigger its own LLM request, and retries multiply that cost again. A trace tree makes this visible by showing parent-child relationships, so the expensive branch stands out and the heaviest child span can be isolated. That matters because a single retrieval step may be pulling far more context than needed, or a function call may be looping because the contract is weak. The mechanism is not cost reporting alone, but cost propagation across the tree, which tells teams which step is actually responsible.

Practical implication: inspect child spans first when a workflow cost looks disproportionate.

Why prompt and model experiments need quality gates

Lower cost is not a useful outcome if it degrades output quality. Prompt edits can trim tokens by removing redundant instructions or oversized examples, while cheaper models can work for simple tasks but fail on edge cases. The right control is not just experimentation, but experimentation tied to evaluation. Side-by-side prompt and model tests should run against real workloads, then quality scoring should block changes that save money by breaking correctness, format, or tool use. That turns optimisation into a release discipline rather than an intuition exercise.

Practical implication: gate cost reductions with evals before they reach production.

NHI Mgmt Group analysis

Span-level observability is the real cost-control boundary for production LLM systems. Once prompts, retrieval, tools, and retries are all part of the same request path, total spend alone becomes too coarse to govern. Cost control moves from budgeting after the fact to understanding which execution path created the bill, which is the only way to make optimisation repeatable. Practitioners should treat trace visibility as a control requirement, not a reporting convenience.

LLM cost governance now has a clear quality-risk trade-off. Cheaper prompts and smaller models often reduce spend, but they can also remove critical context or weaken edge-case performance. That makes cost reduction a release-management problem, because the organisation has to prove that savings do not create silent regressions. Practitioners should align cost changes with evaluation gates and rollback discipline.

Context bloat is becoming a named operational failure mode in AI workflows. System prompts grow, retrieval windows expand, and tool chains multiply until token use becomes disconnected from user value. This is not simply inefficiency, it is governance drift inside the execution path. Practitioners should treat oversized context as a measurable control failure, then trim it with trace evidence instead of assumptions.

AI cost optimisation is increasingly a lifecycle process, not a one-time tuning exercise. Production traffic changes, prompts accrete instructions, and new spans appear as products mature. That means the optimisation model has to be continuous, with production traces feeding back into reusable eval cases. Practitioners should build a feedback loop that preserves the cost-quality balance as the application evolves.

Machine decision pipelines inherit identity-like governance problems when they fan out across tools and models. The more an LLM workflow delegates work to external tools and chained calls, the harder it becomes to attribute responsibility for spend, output quality, and failure points. This is where NHIMG sees the bridge to broader governance: the control problem is not only the model, but the execution identity of the workflow itself. Practitioners should govern these paths as accountable systems, not opaque prompts.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Guide to the Secret Sprawl Challenge

What this signals

Context bloat is the cost signature teams should watch for. As LLM systems mature, prompts, retrieval windows, and tool chains tend to expand faster than governance processes do. That makes trace-level cost review an ongoing operational control, not a one-off optimisation exercise.

Cost governance now overlaps with release governance. If a cheaper prompt or model can degrade output quality, then savings have to be validated before deployment. Teams that separate optimisation from CI/CD will keep discovering regressions after users do.

For practitioners

• Map cost to individual spans Attach estimated cost and token counts to each model call, tool call, and retry path so the expensive step is visible inside the trace rather than hidden in aggregate spend.
• Investigate the heaviest branches first Start with the most expensive production traces and drill into child spans to identify whether prompt bloat, retrieval overreach, or repeated tool calls is driving the bill.
• Trim prompts with real workload evidence Reduce redundant instructions, oversized examples, and system prompt growth only after comparing prompt variants against the same traced inputs.
• Gate cost reductions with evals Run candidate prompt or model changes through quality scoring before release so lower spend does not create silent regressions in accuracy, format, or tool use.
• Convert costly traces into repeatable tests Turn expensive production traces into permanent eval cases so the same failure or waste pattern is checked on future prompt and model changes.

Key takeaways

• LLM cost problems are usually caused by hidden execution details, not just by the choice of model.
• Span-level tracing is the control that exposes where tokens are being spent across prompts, tools, retries, and retrieval.
• The safest cost reductions are the ones that pass quality gates before they reach production.

Key terms

• Span-Level Tracing: Span-level tracing records each step in an LLM request as an individual unit, including model calls, retrieval, tool use, and retries. It lets teams attribute cost and behaviour to the exact action that produced it, which is essential when one request fans out into many hidden subcalls.
• Context bloat: The accumulation of tool definitions, schemas, and metadata in an agent’s working context until reasoning capacity is consumed by exposure rather than task execution. It matters because too many connected tools can reduce performance, obscure intent, and make access governance harder to reason about.
• Approval Gate: An approval gate is the human or policy checkpoint that must be crossed before an AI system can take a state-changing action. For identity governance, the gate is the point where responsibility stays with the organisation, and it should be enforced separately from the system’s ability to observe or recommend.
• Trace Tree: A trace tree is a structured record of an agent session broken into linked events such as prompts, tool calls, model outputs, and artifacts. It lets practitioners see how work unfolded, where tokens were consumed, and which steps produced value or waste.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore the course if your programme needs a clearer model for governing machine identities, access, and lifecycle controls.