Notifications
Clear all
Topic starter
18/08/2026 2:38 pm
TL;DR: Agent optimization can cut latency by 56%, tokens by 27%, and estimated cost by 44%, but also degrade answer completeness until tracing and fixed-dataset evals expose the regression and enable iterative recovery, according to Arize. The lesson is that agent optimisation needs measurement discipline, not transcript intuition, because efficiency gains can conceal quality loss.
NHIMG editorial — based on content published by Arize: How to improve agent skills with tracing and evals
By the numbers:
- The first experiment produced 9,194ms average latency without the skill and 4,081ms with it.
Questions worth separating out
Q: How should teams evaluate agent skill changes without relying on transcript review alone?
A: Teams should compare skill variants on the same model, prompt scaffolding, and task, then score them with a fixed evaluation dataset.
Q: Why do faster agent outputs sometimes make governance worse?
A: Faster outputs can make governance worse when the optimisation removes information that the workflow actually needs.
Q: How do security teams know if agent observability is actually working?
A: Observability is working only when teams can tie together token activity, tool calls, and latency for a specific agent session.
Practitioner guidance
- Implement trace tagging for every agent variant Tag each run with the exact variable under test, such as skill on or off, prompt version, or tool configuration, so comparisons are queryable instead of manual.
- Define a completeness evaluator before optimisation Choose the quality dimension most likely to fail, then score against it on a fixed dataset that reflects real production interactions, not synthetic edge cases.
- Set release gates for quality and efficiency together Require the agent to clear a predetermined completeness threshold while preserving acceptable latency and token use before promoting the change.
What's in the full article
Arize's full analysis covers the operational detail this post intentionally leaves for the source:
- The exact tracing and span-tagging setup used to compare skill-on and skill-off runs.
- The structure of the fixed evaluation dataset and how completeness was scored against it.
- The multi-pass optimisation loop that recovered quality without surrendering the efficiency gains.
- The repository and reproducible workflow for testing the same method against your own agent skills.
👉 Read Arize's analysis of tracing and evals for agent skill optimisation →
Agent skills and evals: are your speed gains hiding regressions?
Explore further
18/08/2026 3:23 pm
Agent skill optimisation creates a new governance debt: behaviour can improve operationally while degrading task integrity. The article shows that speed and cost gains are not sufficient evidence of better agent performance. For agentic AI programmes, the governance question is whether the system still produces complete, reliable outcomes after a behavioural change. Practitioners should treat every skill rewrite as a control change, not a style tweak.
A question worth separating out:
Q: How can organisations stop optimising AI agents for speed at the expense of quality?
A: Set the quality bar before tuning begins, then require the agent to meet that bar on a stable dataset before release. If the change improves latency or cost but lowers completeness, it is not an acceptable production version. Governance should treat quality loss as a failed control, not a tolerable trade-off.
👉 Read our full editorial: Tracing and evals expose when agent skills trade quality for speed
