Notifications
Clear all
Topic starter
20/08/2026 5:51 pm
TL;DR: Coding agents can change prompts, tools, retrieval logic, and application behavior in a single run, but Arize argues that spot checks are not enough once those changes affect multi-step systems. The evaluation harness pattern adds tracing, evaluators, and action loops so teams can catch regressions before they ship, which is increasingly important as AI-assisted development expands.
NHIMG editorial — based on content published by Arize: How to add an evaluation harness to your Gemini CLI coding agent
Questions worth separating out
Q: How should security teams govern coding agents that can act on untrusted input?
A: Treat the agent as a privileged non-human identity and govern it with least privilege, runtime enforcement, and decision lineage.
Q: Why do agent-driven code changes need more than manual spot checks?
A: Manual spot checks miss the full trajectory of an agent run.
Q: What breaks when evaluation is not tied to an action workflow?
A: Telemetry without action creates noise, not control.
Practitioner guidance
- Instrument agent sessions end to end Capture traces, spans, and session IDs for every agent-assisted change so teams can reconstruct the full decision path, not just the final code diff.
- Gate releases on regression thresholds Use evaluator results in CI to block changes when quality, safety, or policy scores fall below an agreed threshold.
- Separate tool authority from review authority Restrict which tools the coding agent can invoke and require a human or automated approval path before any change that affects retrieval, prompts, or access-related configuration.
What's in the full article
Arize's full article covers the operational detail this post intentionally leaves for the source:
- Step-by-step setup for an evaluation harness around Gemini CLI and Arize Skills
- Command-level workflow for exporting traces, creating datasets, and running experiments
- Examples of evaluators for factuality, policy compliance, and regression detection
- MCP integration details for persistent IDE and terminal workflows
👉 Read Arize's guide to adding an evaluation harness to Gemini CLI coding agents →
Coding agents and evaluation harnesses: are your controls keeping up?
Explore further
20/08/2026 6:30 pm
Evaluation debt is becoming a form of AI governance debt: when teams automate code changes without a repeatable way to score the result, they create blind spots that look like productivity gains until regressions appear. The article shows that the problem is not generating the change, but proving the change improved system behaviour. That is the same control gap seen in many fast-moving AI programmes. Practitioners should treat evaluation coverage as a governance requirement, not a nice-to-have.
A question worth separating out:
Q: Why do local AI agents complicate identity and access management?
A: They can retain legitimate permissions while changing timing, prioritisation, and action sequence outside human presence. That means the visible identity may remain stable even as the operational behaviour becomes autonomous. IAM teams then lose the simple link between user session, authorisation, and accountability.
👉 Read our full editorial: Evaluation harnesses are the missing control for coding agents
