Tracing and evals expose when agent skills trade quality for speed

At a glance

What this is: This is an Arize analysis of using tracing and evals to detect and correct quality regressions caused by an agent skill that improved efficiency but reduced completeness.

Why it matters: It matters to IAM and security teams because the same pattern appears in agentic AI governance, where optimisation can hide control failure unless instrumentation proves the change is safe.

By the numbers:

• Enabling the skill reduced token usage by 27%, latency by 56%, and estimated cost by 44%, but also reduced answer completeness.
• The first experiment produced 9,194ms average latency without the skill and 4,081ms with it.
• Average estimated cost per run fell from $0.559 to $0.312 when the skill was enabled.

👉 Read Arize's analysis of tracing and evals for agent skill optimisation

Context

Agent optimisation is a governance problem as much as a performance problem. When a skill changes how an AI agent responds, shorter outputs, lower token use, and faster runtime can look like progress even when the agent quietly omits required information. The article is about that measurement gap, and it is especially relevant wherever agent behaviour affects access decisions, support workflows, or security operations.

In identity and security programmes, the same failure mode shows up when teams trust visible efficiency rather than verified outcomes. For agentic AI, that means tracing, fixed datasets, and quality evaluators become control evidence, not optional tuning aids. The starting position in the article is common: teams see a cleaner transcript and assume the system is better.

The practical lesson extends beyond AI to any workflow where automation can optimise the wrong thing. If a control can make a system faster while degrading completeness or accuracy, the programme needs an evaluator that measures the business-critical dimension directly.

Key questions

Q: How should teams evaluate agent skill changes without relying on transcript review alone?

A: Teams should compare skill variants on the same model, prompt scaffolding, and task, then score them with a fixed evaluation dataset. Transcript review is useful for context, but it misses subtle omissions once outputs look cleaner. The deciding evidence should be traceable metrics for quality, cost, and latency, not visual preference.

Q: Why do faster agent outputs sometimes make governance worse?

A: Faster outputs can make governance worse when the optimisation removes information that the workflow actually needs. A shorter transcript may look better to reviewers, but the underlying task can become less complete or less reliable. That is why teams need task-specific evaluators, not just performance metrics, when changing agent behaviour.

Q: How do security teams know if agent observability is actually working?

A: Observability is working only when teams can tie together token activity, tool calls, and latency for a specific agent session. If those signals are disconnected, the logs may show traffic but not governance. The test is whether investigators can reconstruct the path of a meaningful agent action from start to finish.

Q: How can organisations stop optimising AI agents for speed at the expense of quality?

A: Set the quality bar before tuning begins, then require the agent to meet that bar on a stable dataset before release. If the change improves latency or cost but lowers completeness, it is not an acceptable production version. Governance should treat quality loss as a failed control, not a tolerable trade-off.

Technical breakdown

How tracing makes agent skill changes comparable

Tracing turns agent changes into measurable variables rather than subjective impressions. By tagging spans with attributes such as skill enabled or disabled, teams can compare runs on the same model, prompt scaffolding, and task while isolating the effect of one change. OpenInference-style metadata is useful because it preserves context at the span level, which is where behavioural differences often appear. Without that structure, reviewers end up comparing transcripts by feel, which favours cleaner-looking outputs even when they omit required content.

Practical implication: tag every experimental variable in traces so quality, cost, and latency can be compared on the same dataset.

Why completeness evals catch regressions that transcript review misses

A completeness evaluator checks whether a response covers the information a task requires, not whether it reads well. That distinction matters because shorter outputs often look better to human reviewers while silently dropping essential facts. In the article, the evaluator was purpose-built for the task rather than borrowed from a generic benchmark, which is important because quality criteria vary by workflow. For AI governance, this is the difference between style optimisation and safe behaviour validation.

Practical implication: define the quality dimension at risk before tuning an agent, then evaluate that dimension directly.

How a long-running optimisation loop improves skills safely

A long-running agent can revise a skill against a fixed evaluation set and iterate until the quality score improves without giving back all efficiency gains. This is essentially closed-loop optimisation: run, score, revise, and rerun until the target threshold is met. The article shows why one pass is rarely enough. Early revisions overcorrected and later passes recovered balance. For organisations, the key architectural point is that optimisation should be governed by a stable benchmark, not by one-off transcript inspection.

Practical implication: use fixed evaluation datasets and repeatable scoring gates before promoting an agent skill into production.

NHI Mgmt Group analysis

Agent skill optimisation creates a new governance debt: behaviour can improve operationally while degrading task integrity. The article shows that speed and cost gains are not sufficient evidence of better agent performance. For agentic AI programmes, the governance question is whether the system still produces complete, reliable outcomes after a behavioural change. Practitioners should treat every skill rewrite as a control change, not a style tweak.

Tracing is now an evidence layer for AI governance, not just an observability convenience. Tagging each run with the active skill state made the comparison auditable and repeatable. That is the kind of structure AI RMF and agentic security frameworks expect, because unmanaged behavioural drift is a model risk problem. The practical conclusion is that every meaningful agent change needs traceable provenance.

Completeness is the named concept security teams should borrow from this experiment. Completeness loss is the failure mode where an agent becomes terser, cheaper, and more efficient while omitting mandatory information. That is especially dangerous in agentic workflows that support access, triage, verification, or compliance decisions. Teams should define completeness thresholds before optimisation begins and treat falling below them as a release blocker.

Closed-loop evaluation is the only reliable way to tune agent skills at scale. Manual inspection catches obvious regressions but misses subtle omissions once outputs look cleaner. A fixed dataset, repeatable scoring, and iterative revision create a better control loop for AI operations. The broader lesson is that agent governance needs measurement discipline equal to the speed of agent change.

This pattern intersects with identity governance whenever agents make or support decisions that influence access, trust, or workflow approval. If an AI agent is shaping security actions, reduced completeness can mean reduced control fidelity. IAM and AI governance teams should align skill changes with policy intent, evaluator design, and sign-off criteria before deployment.

What this signals

Completeness drift is the control signal most agent programmes still miss. If your evaluation stack only measures latency and cost, you can easily approve a change that makes the system look better while making it less trustworthy. For security and IAM-adjacent workflows, that is the same mistake as trusting a credential inventory without checking whether access still matches policy.

The operating model should shift from transcript inspection to release governance. Teams need fixed datasets, repeatable scoring, and evidence that the new behaviour still satisfies the workflow requirement before it reaches production. That pattern aligns with NIST AI Risk Management Framework thinking, where measurement is part of risk treatment rather than an afterthought.

When agents influence access, verification, or case handling, the relevant question is not whether they are faster. It is whether the faster version still preserves decision completeness, auditability, and policy intent. That is where a well-governed completeness threshold becomes part of the programme's control set.

For practitioners

• Implement trace tagging for every agent variant Tag each run with the exact variable under test, such as skill on or off, prompt version, or tool configuration, so comparisons are queryable instead of manual.
• Define a completeness evaluator before optimisation Choose the quality dimension most likely to fail, then score against it on a fixed dataset that reflects real production interactions, not synthetic edge cases.
• Set release gates for quality and efficiency together Require the agent to clear a predetermined completeness threshold while preserving acceptable latency and token use before promoting the change.
• Use iterative review for agent skills that shorten output Let a long-running optimisation loop revise the skill, rerun the benchmark, and stop only when the new version meets the quality bar without undoing efficiency gains.

Key takeaways

• Agent skills can improve speed and cost while quietly degrading the quality the workflow actually needs.
• Tracing and fixed-dataset evals are what turn behavioural changes into auditable control decisions.
• Completeness thresholds should gate agent releases whenever optimisation could remove required information.

Key terms

• Agent Skill: A reusable package of task-specific knowledge and procedures that an autonomous agent can load when needed. In practice, it separates general awareness from operational detail, which makes enterprise context easier to govern than a single oversized prompt.
• Completeness Evaluator: A completeness evaluator measures whether a response includes the information required for the task. It is more specific than generic quality scoring because it tests coverage against predefined criteria, making it useful when shorter outputs risk omitting essential content.
• Trace Tagging: Trace tagging is the practice of attaching structured metadata to each run or span so experiments can be compared reliably. For AI agents, it creates provenance for prompt versions, skill states, and tool paths, which supports reproducible evaluation and release control.
• Fixed Evaluation Dataset: A fixed evaluation dataset is a stable set of representative interactions used to score changes consistently over time. It reduces guesswork by ensuring that each version of an agent is tested against the same task conditions and quality expectations.

What's in the full article

Arize's full analysis covers the operational detail this post intentionally leaves for the source:

• The exact tracing and span-tagging setup used to compare skill-on and skill-off runs.
• The structure of the fixed evaluation dataset and how completeness was scored against it.
• The multi-pass optimisation loop that recovered quality without surrendering the efficiency gains.
• The repository and reproducible workflow for testing the same method against your own agent skills.

👉 Arize's full post shows the trace setup, evaluator design, and optimisation loop behind the results.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, secrets management, and workload identity. It helps practitioners translate identity controls into measurable operational discipline across modern automation programmes.