Notifications
Clear all
Topic starter
20/08/2026 5:50 pm
TL;DR: The lowest-cost AI agent design is not the cheapest model, but the control logic that balances routing, retries, fallback, and escalation, according to Braintrust. The strongest policies resolved about 93% of tickets at lower cost per resolved request than the always-frontier baseline, while cheap-only strategies underperformed despite lower token spend.
NHIMG editorial — based on content published by Braintrust: How to test agent cost-efficiency with Braintrust
By the numbers:
- When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.
Questions worth separating out
Q: How should security teams govern AI agents that can change actions at runtime?
A: Security teams should govern runtime AI by correlating identity, data, and intent before trusting an action path.
Q: Why do AI agents create new risk in non-human identity management?
A: AI agents create risk because they operate as software identities with delegated authority, but many organisations do not track them with the same discipline applied to users or service accounts.
Q: What do teams get wrong when they measure AI agent cost?
A: They often stop at token price, which hides retries, fallback calls, judge overhead, tool execution, and human cleanup.
Practitioner guidance
- Implement cost-per-resolved-request reporting Track total spend against resolved outcomes, including retries, fallbacks, tool calls, and human interventions.
- Instrument escalation and fallback paths Log every branch decision in the agent workflow so you can see when the system escalates, retries, or abstains.
- Add quality and safety gates to evals Require tool correctness, no unsupported claims, complete key-point coverage, and safe action handling before a run counts as resolved.
What's in the full article
Braintrust's full blog covers the operational detail this post intentionally leaves for the source:
- Policy-by-policy comparison of model routing, retries, and fallback behaviour across the test set
- Per-policy cost-per-ticket and cost-per-resolved-request breakdowns for each control strategy
- Workflow logging and evaluation setup used to score tool correctness, safety, and completeness
- Loop-based frontier analysis showing where each policy sits on the cost and quality curve
👉 Read Braintrust's analysis of agent cost-efficiency strategies and evaluation results →
AI agent cost-efficiency: why routing beats simple model swaps?
Explore further
20/08/2026 6:29 pm
Agent cost governance is becoming a runtime control problem, not a model-selection problem. Once an agent can route, retry, and escalate based on context, the organisation is governing behaviour, not just output. That shifts accountability toward policy design, observability, and exception handling. For IAM and NHI teams, the lesson is that runtime authority needs the same scrutiny as static permissioning.
A question worth separating out:
Q: What should organisations do before giving agents broader tool access?
A: They should validate the control logic with evals that test unsupported claims, unsafe actions, and failed tool calls. If a policy cannot maintain output quality under failure conditions, broader tool access only increases blast radius. Scope should expand only after the escalation path is proven and observable.
👉 Read our full editorial: AI agent cost control depends on routing, not cheaper models
