Evaluation-driven AI agents need specifications, evals and guardrails

At a glance

What this is: This is Arize’s analysis of how evaluation-driven development helps AI agents move from pilot to production without losing control of quality, cost, or risk.

Why it matters: It matters to IAM and security teams because agentic systems need governed identity, access, and review controls before they can safely act in production workflows.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.

👉 Read Arize’s analysis of evaluation-driven development for AI agents

Context

Enterprise AI programmes fail when delivery speed outruns the controls needed to evaluate, monitor, and govern what the system is doing. In agentic workflows, the problem is not just model quality, but the surrounding control plane for specifications, access, and release decisions. That is why AI agent production readiness now intersects directly with IAM, PAM, and NHI governance.

Arize’s discussion is useful because it frames production as an engineering and governance problem, not a demo problem. As AI agents gain more ability to generate code, call tools, and move across systems, the organisation needs evidence-based release gates, scoped permissions, and auditable intervention paths. The same lesson applies to any programme where human and machine actions blend into a single operational workflow.

Key questions

Q: How should security teams govern AI agents that can choose tools at runtime?

A: Security teams should govern runtime agent choice as an access event, not as a simple application action. That means scoping permissions to the task, limiting token lifetime, logging every tool decision, and blocking the agent from reaching systems outside its approved context. Static roles alone are not enough when the execution path changes on each run.

Q: Why do AI agents complicate existing IAM and PAM controls?

A: AI agents complicate IAM and PAM because they often inherit delegated credentials, operate across multiple systems, and keep acting after the initial approval moment has passed. Human session assumptions, periodic reviews, and static privilege models do not reflect that behaviour. The result is a governance gap between what was granted and what the agent can actually do.

Q: What breaks when AI evaluation is added only after a pilot is already working?

A: The team usually discovers that it cannot prove the system is safe, repeatable, or cost-effective enough to scale. Late evaluation often exposes missing edge cases, unclear acceptance criteria, and no agreed rollback path. By then, the pilot has already created expectations, so the organisation is forced to choose between pausing momentum and accepting unmeasured risk.

Q: Who is accountable when an AI agent takes an unsafe action?

A: Accountability should sit with the business owner of the agent, the team that provisioned the access, and the control owners responsible for monitoring and revocation. If no one can answer who approved the identity, the scope, and the oversight model, the governance framework is not complete enough for production.

Technical breakdown

Why pilot purgatory happens in agentic AI delivery

Pilot purgatory appears when a system can impress in a demo but lacks the operational controls required for repeatable production use. In practice, that means missing specifications, weak evaluation criteria, no observable release gate, and no dependable ownership for access or rollback. Agentic AI intensifies the gap because the system can produce output faster than the surrounding process can validate it. The result is not just a delayed launch, but a widening mismatch between machine pace and organisational readiness. When teams treat the model as the main constraint, they miss the real bottleneck: the control environment around the model.

Practical implication: define the release criteria, ownership, and monitoring model before the agent is allowed to act.

How evaluation harnesses turn prompts into governable systems

An evaluation harness is the set of tests, golden cases, and scoring rules that determine whether an agent is safe and useful enough to ship. It converts vague intent into evidence by checking whether outputs match the business task across known scenarios, edge cases, and failure thresholds. For agentic systems, this matters because behaviour can be plausible line by line while still producing unsafe actions across a workflow. The harness also becomes a feedback loop in production, where new traces and failures are added back into the test set. That turns evaluation into a living control rather than a one-time launch checklist.

Practical implication: build regression suites around the decisions and tool calls that would be unacceptable if they repeated in production.

Why cost-per-outcome matters more than token spend

Token cost is only a partial view of AI economics because it ignores rework, human review, retries, exceptions, and fallback handling. Cost-per-outcome measures the full cost of reaching a usable result, which is what matters when an agent sits inside a business process. A cheaper model can still be more expensive if it generates outputs that require repeated correction or manual escalation. This matters for governance as well as finance, because the same metric reveals whether autonomy is creating value or simply shifting labour downstream. In production, the right unit is the completed business outcome, not the raw inference bill.

Practical implication: track each agent against a business outcome metric such as cycle time, error rate, or time to resolution.

NHI Mgmt Group analysis

Evaluation-driven development is becoming the control plane for agentic AI, not just a software practice. The article shows that speed is now being gated by the quality of the surrounding evidence system, including specs, evals, observability, and rollback. That is exactly how agentic AI changes governance: access and action must be justified by measurable behaviour, not by the promise of the model. For identity programmes, the implication is direct. When agents can call tools and move across workflows, their permissions and review boundaries need the same discipline as any other privileged system.

AI agent governance has a new named failure mode: pilot purgatory. This is the state where a system proves capability in a demo but cannot cross into production because the organisation has no durable way to evaluate, monitor, or own it. The problem is not model weakness alone. It is governance debt accumulating around undocumented intent, unclear accountability, and unmeasured side effects. Practitioners should treat long-lived pilots as evidence that the control model is incomplete, not as proof that the use case is unviable.

Agent autonomy without scoped identity is just delegated risk at machine speed. The article’s focus on access, rollback, and human review aligns with a core NHI concern: an AI system that can act needs a bounded identity, not just a prompt and a toolchain. That means permissions, audit trails, and intervention points must be designed together. In IAM terms, the issue is not whether the agent can do the work, but whether the organisation can constrain, observe, and revoke that work when behaviour changes.

Cost governance now belongs inside AI release governance. Turner and Khare’s emphasis on cost-per-outcome shows that technical success is not enough if the workflow becomes economically unstable at scale. For the field, this shifts AI governance away from abstract risk language and toward measurable operating thresholds. Teams that cannot explain the cost of retries, review, and exceptions do not yet have production control. The practitioner conclusion is simple: if the economics are ungoverned, the autonomy is ungoverned.

Specifications are becoming machine-readable policy for AI delivery. The spec no longer serves only as documentation for humans. It is now a control artifact that shapes what the agent is allowed to infer, change, and test. That matters because specifications create a shared contract for product, engineering, and evaluation teams, while also giving reviewers a stable basis for approval. Practitioners should treat specification quality as a security and governance issue, not just a delivery discipline.

What this signals

Pilot governance is now a security control, not a delivery milestone. When AI agents can change code, invoke tools, or move data, the first question is no longer whether the demo works. The question is whether the programme has a measurable release gate, a revocation path, and a traceable decision record. That is where NIST AI Risk Management Framework discipline and access governance start to converge.

Agent identity needs to be managed like any other privileged workload. If an AI system can take action, it needs a defined identity, scoped permissions, and production monitoring that matches the risk of its actions. That aligns with the control logic behind the OWASP Agentic AI Top 10, especially where tool misuse and privilege abuse can turn a useful agent into an unmanaged operator.

Cost visibility will separate serious production programmes from experiments. Teams that cannot explain cost per outcome, rollback conditions, and exception handling will struggle to justify autonomy at scale. For identity leaders, the lesson is that governance now spans both access scope and economic blast radius, because a misrouted agent can burn budget and control confidence at the same time.

For practitioners

• Require release criteria before agent deployment Tie each AI agent to explicit entry and exit criteria, including the evidence needed to move from pilot to production, the owner who approves release, and the fallback state if behaviour drifts. This should include access scope, rollback conditions, and the monitoring signals that would force a pause.
• Build evaluation harnesses around high-risk actions Create golden datasets and regression tests for tool use, data access, and workflow completion, especially where an agent can trigger downstream changes. Include cases that would be unacceptable if they repeated in a live environment, not just cases that look good in a demo.
• Bind agent autonomy to auditable identity controls Issue each agent a bounded identity with least-privilege access, traceable actions, and defined human intervention paths. Connect access reviews to the behaviours the agent actually performs, so permissions can be revoked or narrowed when usage changes.
• Measure cost per outcome, not token consumption Track the business result that the agent is supposed to improve, then compare it with the full cost of retries, review, exceptions, and rework. Use that metric in release decisions so finance, engineering, and security can see whether the workflow is becoming more or less controllable.
• Instrument production traces for governance review Capture the prompt, retrieved context, tool calls, outputs, and reviewer decisions that led to each material action. Those traces support incident triage, policy tuning, and compliance review when an agent behaves outside its intended scope.

Key takeaways

• AI agent delivery now depends on governed evidence, not just model capability, because production failures often come from the surrounding control system.
• The strongest signal in the article is that evaluation, observability, and rollback have become prerequisites for scale, especially where agents can take consequential actions.
• Security and IAM teams should treat agent identity, scoped access, and auditability as part of release engineering, not as an afterthought.

Key terms

• Evaluation Harness: An evaluation harness is the test environment, scripts, permissions, and network setup used to assess a model or system before release. Its security matters because the harness is often the difference between harmless testing and a real-world compromise path if it is allowed to reach external systems.
• Cost per outcome: A measure of the full cost required to complete a business task, including retries, review, exceptions, and rework. It is more useful than token cost alone because it reflects whether an AI workflow is actually efficient at scale.
• Agent Identity: An agent identity is the set of attributes, credentials and permissions assigned to an autonomous software entity. It is treated as a non-human identity because it can authenticate, act on systems and accumulate access over time, which creates governance, audit and lifecycle obligations similar to other production identities.
• Pilot Purgatory: A delivery state where AI initiatives cannot move from experimentation to production because identity, approval, and compliance steps are handled manually. In practice, the programme gets stuck between innovation pressure and governance delay, which encourages workarounds, shadow deployments, and shared credentials.

What's in the full article

Arize's full analysis covers the operational detail this post intentionally leaves for the source:

• The exact evaluation and release workflow CVS Health used to move agents from prototype to production.
• Examples of specification, harness, and observability patterns that support repeatable AI delivery.
• The cost and KPI framing used to decide whether an agent is actually worth scaling.
• Practical guardrails and review checkpoints for production AI workflows.

👉 The full Arize discussion covers the CVS Health examples, production harness design, and governance mechanics.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and agentic AI identity. It gives practitioners a structured way to connect access control, lifecycle discipline, and production governance.