Guardrails and evals are both required for reliable AI agents

At a glance

What this is: This is an analysis of why AI agent reliability depends on both guardrails and evals, with the key finding that failures often come from orchestration and control design rather than model capability.

Why it matters: For IAM and security practitioners, the article matters because agent autonomy turns tool access, approval boundaries, and execution control into governance problems that resemble privileged access management for software actors.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.

👉 Read Arize's analysis of AI agent guardrails and evals

Context

AI agent reliability is now a governance problem, not just a model-quality problem. When an agent can call tools, manage state, retry tasks, and decide when a workflow is complete, the real failure surface moves into orchestration, permissions, and recovery logic. That is why the article’s primary keyword, AI agent guardrails, belongs in the same conversation as evals, access boundaries, and runtime control.

The article treats agent behaviour as a system property. That matters for identity and access programmes because every additional tool, credential, and approval path creates a new policy boundary that must be enforceable in code, observable in traces, and measurable in evaluation. For teams already managing workload identity or non-human access, this is the same control problem in a more dynamic form.

Key questions

Q: How should security teams use static guardrails for AI agents?

A: Use static guardrails as a first-pass control for known bad inputs, prohibited outputs, and obvious data leakage. Then pair them with tool restrictions, runtime policy checks, and logging, because fixed rules cannot reliably handle indirect prompt injection or context-dependent abuse. The control is useful, but it is only one layer in a broader agent governance model.

Q: Why do autonomous agents create a bigger governance problem than chat-based AI assistants?

A: Chat-based assistants usually begin with a visible human prompt, so the initiation point is easier to govern and audit. Autonomous agents can start from external events, which means the triggering condition, payload, and connector become part of the identity attack surface. That expands the control problem beyond prompt content alone.

Q: What do security teams get wrong about AI agent benchmarks?

A: They often measure end-to-end complexity or general model quality instead of the exact failure moment. The result is a score that looks useful but does not isolate whether the backbone, tool access, or orchestration caused the problem. Benchmarks need a specific state, attack vector, and scoring function to be operationally meaningful.

Q: Who is accountable when an AI agent acts outside its intended scope?

A: The organisation is accountable, but operational responsibility should sit with a named owner and a governance process that can explain the agent’s purpose, access, and recorded actions. Without that, autonomous behaviour becomes unassignable risk rather than managed automation.

Technical breakdown

AI agent guardrails vs evals: different control layers

Guardrails and evals solve different problems. A guardrail is an enforced constraint at runtime, such as limiting tool access, response concurrency, retries, or approval thresholds. An eval is a judgment after the fact about whether the agent’s behaviour was correct, safe, grounded, or useful. The key architectural point is that an eval can detect failure, but it cannot stop a prohibited action from happening. That is why long-running agents need a harness that coordinates permissions, state, limits, and feedback, not just a model prompt.

Practical implication: put irreversible actions behind code-enforced policy, not post-run review.

Why long-running agent workflows increase failure surface

A short chatbot exchange has one output path. A longer agent run can retrieve data, call tools, modify files, branch, retry, delegate, and decide when to stop. Each step creates a new chance for drift, including wrong tool arguments, stale context, conflicting branches, or wasteful retries. In practical terms, the more autonomy you grant, the more you must treat the agent as an execution system with state transitions, not as a single inference event. This is where execution tracing and permission design become essential.

Practical implication: map each agent step to an explicit control point, especially where tools can change state or data.

Why evaluator quality depends on context and recency

An evaluator is only as useful as the context it can see. If it grades outputs without the current date, the retrieved sources, the workflow policy, or the user’s actual goal, it may score a correct result as wrong or miss a harmful one that sounds fluent. The article shows that evaluation is not universal. A customer-support reply, a code migration, and a financial report each need different evidence. Reliable eval design therefore requires domain-specific criteria and the same skepticism teams apply to production logic.

Practical implication: test evaluators with current, domain-specific evidence before using their scores to gate releases.

Threat narrative

Attacker objective: The objective is not traditional exfiltration but uncontrolled or misleading agent behaviour that causes operational, compliance, or decision risk.

1. Entry occurs when an agent is allowed to run with broad tool access and insufficient runtime constraints, letting a harmless-looking task expand into multi-step execution.
2. Escalation happens when the agent branches, retries, or reuses stale context, creating overlapping actions or unintended side effects across tools and state.
3. Impact appears when the workflow produces unsafe, confusing, or incorrect outcomes that cannot be trusted because the control harness failed to contain the run.

NHI Mgmt Group analysis

Guardrails are the control plane for agent authority, not a polishing layer. The article is right to separate runtime constraints from post-run evaluation because the two controls answer different governance questions. For IAM teams, that means the meaningful question is not whether an agent can be scored after execution, but whether it was ever permitted to take the risky action in the first place. In identity terms, this is least privilege for agent behaviour, enforced at the action layer.

AI agent governance is converging with non-human identity governance. Once an agent can call tools, hold credentials, and act for hours, it behaves like a privileged software identity with variable intent. That makes agent authorization, approval, and revocation lifecycle issues, not only model issues. Organisations that already struggle with service account sprawl should recognise the same pattern here, only with faster decision cycles and broader tool reach.

Harness failure is the named concept practitioners should watch. The article shows that many dangerous outcomes come from the harness, the execution environment that coordinates tools, state, retries, and permissions. When the harness allows parallel responses, stale context, or unconstrained retries, the model may be competent while the system still fails. Practitioners should therefore evaluate the harness as a governance object, not just the model.

Evals without bounded evidence are dangerous because they can certify the wrong thing. A score that ignores recency, workflow state, or business context can create false confidence and mask operational drift. This is especially relevant where AI systems make or assist decisions that affect access, data handling, or external communications. Teams should treat evaluation design as a control assurance function, not a reporting exercise.

Agent autonomy changes the security contract, which is why identity teams must get involved early. The more time an agent is allowed to run, the more important approval rules, traceability, and bounded delegation become. That pushes agent design into the same conversation as PAM, workload identity, and policy enforcement. Practitioners should align AI engineering with identity governance before autonomy expands beyond reversible actions.

What this signals

Harness failure: the next wave of AI risk is less about model jailbreaks and more about execution systems that permit too much authority for too long. As agents move from short prompts to multi-hour workflows, teams need controls that are observable, testable, and revocable at runtime. Identity teams should treat delegated tool access as a governance boundary, not a convenience feature.

The practical signal for programmes is that agent control maturity will be measured by traceability, approval logic, and runtime containment rather than by model accuracy alone. Organisations that already struggle to govern service accounts, API tokens, and privileged automation will find the same issues reappearing in agent systems, but with faster change cycles and more complex decision paths.

For practitioners

• Define executable guardrails for agent authority Convert prompt intent into code-enforced limits on tool access, response concurrency, retry counts, and approval requirements for irreversible actions. This is the control layer that stops unsafe behaviour before it happens, especially when an agent can modify data or communicate externally.
• Separate reversible tasks from privileged actions Classify agent actions by business impact, then require human approval for actions that spend money, change production data, send external messages, or create legal commitments. That separation is the practical boundary between experimentation and governed operation.
• Instrument the full execution trajectory Retain traces for tool calls, branching decisions, retries, and context changes so reviewers can reconstruct how the agent reached an outcome. Without trajectory evidence, a final answer can hide policy violations or wasteful execution paths.
• Design evals around current context Give evaluators the sources, policies, timestamps, and user goals they need to judge the run correctly. A judge that works from stale knowledge will produce false negatives and false confidence, which is especially risky in time-sensitive workflows.

Key takeaways

• AI agent reliability depends on the harness, because orchestration and permissions can fail even when the model understands the task.
• Longer agent runs widen the failure surface, so runtime controls matter more as autonomy increases.
• Security teams should pair guardrails with evals, because scoring behaviour after execution does not prevent unsafe actions during execution.

Key terms

• Guardrails: Guardrails are policy controls that inspect prompts and model outputs against defined safety, privacy, and compliance rules. In AI operations, they reduce harmful language and disclosure risk, but they do not replace entitlement management, logging, or identity governance for the systems that call the model.
• Evals: Evals are structured measurement systems for judging whether an AI tool performs well across many runs. They do not try to prove one exact output is correct. Instead, they use scoring, thresholds, and repeated scenarios to show whether the tool is reliably useful in practice.
• Harness: The harness is the layer of instructions, policies, and approval logic wrapped around an AI agent. It is where organisations try to constrain behaviour, but it only works if the rules are explicit, current, and enforced outside the model itself.
• Agent Authority: The permission an AI agent receives to act on behalf of a verified person. In this model, authority is inherited rather than original, so governance must trace the agent back to the human intent, device context, and current trust state that authorised it.

What's in the full article

Arize's full post covers the operational detail this post intentionally leaves for the source:

• Concrete examples of how guardrails are encoded in the agent harness for tool access, retries, and response concurrency.
• The evaluator design questions that determine whether an eval can actually judge a run using current context and policy.
• A step-by-step feedback loop showing how failed runs become prompts, tests, and code changes for the next iteration.
• The voice-agent failure example that illustrates why output quality alone does not prove safe execution.

👉 The full Arize article covers harness design, evaluator context, and failure-to-fix workflows in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management through a practitioner lens. It is a fit for security teams that need to govern autonomous software, delegated access, and identity controls across modern environments.