Notifications
Clear all
Topic starter
20/08/2026 5:58 pm
TL;DR: A Claude Code agent improved an AI newsletter generator fastest when the team fixed data, prompts, and evaluators in sequence, not just model output, and a better metric for coverage outperformed link counting, according to Arize. The lesson for agentic AI governance is clear: control the objective function, because agents will optimise the hill you point them at.
NHIMG editorial — based on content published by Arize: How We Used Evals (and an AI Agent) to Iteratively Improve an AI Newsletter Generator
By the numbers:
- 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
Questions worth separating out
Q: What breaks when an AI agent is optimised against the wrong eval metric?
A: The agent learns to satisfy the metric rather than the real objective.
Q: Why do AI coding agents create governance risk even when they improve productivity?
A: They create risk because faster output does not guarantee safer output.
Q: How can teams tell whether an agentic SOC workflow is actually under control?
A: Look for three signals: every action is attached to a readable case, permissions are narrow and role-specific, and changes are versioned rather than edited ad hoc.
Practitioner guidance
- Define outcome-based evals for agentic workflows Measure whether the generated output is faithful, complete, and usable in context, not just whether a mechanical check passes.
- Normalise source data before agent execution Replace shortened links, standardise metadata, and remove ambiguous artefacts before passing content to the model or agent.
- Separate prompt quality from control quality Treat prompts as behaviour shaping, but treat access, coverage, and auditability as control problems.
What's in the full article
Arize's full article covers the operational detail this post intentionally leaves for the source:
- The exact eval definitions used for faithfulness, structure adherence, hallucinated links, and content coverage.
- The experiment-by-experiment changes to lib/newsletter.ts and evals/newsletter-eval.ts that moved the scores.
- The reasoning behind replacing a link-counting metric with an LLM judge for tweet coverage.
- The open-source workflow and Phoenix experiment history behind the agent-led iteration loop.
👉 Read Arize's analysis of eval-driven AI agent development for the newsletter generator →
AI agents and evals: what changes for practitioners now?
Explore further
20/08/2026 6:37 pm
Proxy metrics become a security issue once agents can optimise them autonomously. The article shows how easily a system can satisfy a mechanical evaluator while degrading the actual user experience. In AI governance terms, that is evaluation drift, and in identity terms it is the same structural risk as overbroad delegated access: the system does exactly what the metric rewards, not what the programme intended. Practitioners should treat evaluation design as a control surface, not a reporting afterthought.
A question worth separating out:
Q: Should organisations allow AI agents to act on production data before evals are mature?
A: Only with tightly bounded permissions and a clear rollback path. Mature evals should check not just correctness, but whether the agent stayed within approved data, tool, and action boundaries. If those checks are missing, the agent can scale mistakes faster than humans can review them.
👉 Read our full editorial: Eval-driven development exposes where AI agents help and fail
