Notifications
Clear all
Topic starter
20/08/2026 5:27 pm
TL;DR: AI agents improve less through better prompts than through a better harness around the model, according to Arize, where traces, targeted evals, failed-span review, and safe change gates turn opaque behaviour into a repeatable engineering loop. The governance lesson is that observability, policy, and review now matter as much as model quality for reliable agent systems.
NHIMG editorial — based on content published by Arize: How to build a better agent harness with traces and evals
Questions worth separating out
Q: How should security teams govern AI agents that can choose tools at runtime?
A: Security teams should govern runtime agent choice as an access event, not as a simple application action.
Q: Why do traces matter more than final outputs for agent oversight?
A: Final outputs tell you whether the answer looked right, but traces show where the agent’s reasoning, retrieval, or tool use broke down.
Q: What do security teams get wrong about evaluator design for agents?
A: They often treat evaluators as fixed infrastructure instead of versioned instruments that can drift.
Practitioner guidance
- Instrument every agent step Capture tool calls, retrieved context, intermediate outputs, and final responses so the full decision path is replayable during review.
- Build span-level evals from real failures Start with one behaviour that traces show is breaking, such as wrong source selection or bad scoring, and evaluate that span directly.
- Separate agent defects from evaluator defects Review a small sample of failures and passed cases to decide whether the agent, the rubric, or the evaluator is wrong before changing anything.
What's in the full article
Arize's full article covers the operational detail this post intentionally leaves for the source:
- The demo workflow for building a PM agent that scores GitHub issues, discussions, and releases.
- How traces are turned into targeted evals for specific spans rather than only report-level scoring.
- The failure-review loop used to decide whether the agent, rubric, tool chain, or evaluator is wrong.
- The distinction between low-risk and high-risk changes when a proposed fix affects production behaviour.
👉 Read Arize's analysis of building a better agent harness with traces and evals →
Agent harness traces and evals: what teams need to change?
Explore further
20/08/2026 6:16 pm
Agent harnesses are becoming the control plane for AI behaviour. The article shows that model quality alone does not explain reliable agent performance. Traces, evals, tool selection, context boundaries, and review gates together determine whether an agent can be improved safely. For AI governance teams, that makes the harness the place where policy becomes operational, especially when agents can act across tools and workflows.
A question worth separating out:
Q: How do teams know when an agent change is safe to ship?
A: A change is safer when the trace shows the exact failure, the eval measures the right behaviour, and a human reviews any update that changes tool access, routing, prompts, or quality criteria. If any of those parts are unclear, the organisation is changing the system without fully understanding the control impact.
👉 Read our full editorial: Agent harness traces and evals are the real improvement loop
