Guardrails and evals are both required for reliable AI agents

At a glance

What this is: Arize argues that reliable AI agents need both guardrails and evals because model quality alone cannot control orchestration failures, conflicting actions, or weak recovery logic.

Why it matters: IAM, NHI, and AI security teams should treat agent reliability as a governance problem over permissions, tool use, and approval boundaries, not just a model-quality problem.

Context

AI agents create a wider failure surface than chatbots because they can call tools, manage state, retry work, and decide when a task is complete. That makes reliability dependent on the system around the model, not just the model itself, and it is where identity, permissions, and approval boundaries start to matter.

The article’s core point is that guardrails and evals solve different governance problems. Guardrails enforce what an agent is allowed to do, while evals measure whether the run produced an acceptable outcome; in agentic systems, both are required because a good final answer can still come from an unsafe or wasteful path.

Key questions

Q: How should enterprises enforce boundaries for AI agents that can act on tools and data at runtime?

A: Enterprises should place an independent enforcement layer between the agent and the tools it uses. The layer must evaluate each action against live context, including the agent’s identity, the customer policy, the target data, and the intended operation. If policy says no, it blocks the action regardless of what the model was prompted to do or what the agent believes is allowed.

Q: Why do reliable agent systems need both guardrails and evals?

A: Because they answer different questions. Guardrails decide what an agent is allowed to do during execution, while evals judge whether the resulting behaviour met the intended standard. One prevents unsafe actions, the other measures quality after the fact, so removing either leaves a gap in agent governance.

Q: What are the signs that an AI agent harness is not working?

A: Look for rising token cost, tool calls that surprise the team, traces that are hard to interpret, and failures that only appear after rollout. If evals pass in a notebook but break once real users interact with the agent, the harness is not capturing production behaviour. That usually means the control boundary is too loose or the evaluator set is too shallow.

Q: How do organisations make eval scores useful for production decisions?

A: Give evaluators the context they need, including the retrieved sources, current policy, user goal, and full execution trace. Then use the score as one input to release decisions, not as proof of safety. If the judge cannot see the operating conditions, its result is only a partial signal.

Technical breakdown

Why long-running agent runs fail differently from chatbots

A chatbot usually has one exchange pattern: input, model output, done. A long-running agent can retrieve data, call tools, modify files, retry failures, maintain state, and branch into parallel actions. Each step adds a new failure mode, including stale context, conflicting branches, wrong tool arguments, and loops that spend time or tokens without moving the task forward. The reliability problem is therefore orchestration, not only generation. The harness determines whether the model’s output becomes a controlled workflow or a sequence of uncontrolled actions.

Practical implication: Practitioners need run-level controls, traceability, and concurrency limits before they expand agent autonomy.

Guardrails as executable constraints on agent authority

Guardrails are code-level constraints that define what an agent may do, which makes them different from prompt instructions or outcome scoring. They can limit tool access, cap parallel responses, require approval for external actions, and stop a workflow when boundaries are crossed. In identity terms, this is where authorization becomes operational: an agent may be technically capable of acting, but still not permitted to use every tool or credential it can reach. That distinction is central to safe agent design because many failures are permission failures, not intelligence failures.

Practical implication: Map agent permissions to explicit allowlists, approval gates, and task-scoped tool access rather than relying on prompts alone.

Why evals can misread the right answer for the wrong run

Evals judge behavior after the fact, but they are only as good as the evidence they can see. A generic evaluator may score a current report incorrectly if it lacks the run’s retrieved sources, the task context, the policy constraints, or the user’s actual goal. That means an apparently objective score can still reflect stale knowledge or the wrong success criteria. Evals are therefore a measurement system, not a control system, and they should be designed with the same discipline as production policy enforcement.

Practical implication: Give evaluators the run context they need and treat their scores as evidence, not as a substitute for enforcement.

NHI Mgmt Group analysis

Agent reliability is now a governance problem, not just a model-quality problem. As agent runtimes lengthen, the real risk shifts from whether the model can answer to whether the system can contain the actions that follow. That pushes teams toward explicit authority boundaries, traceable execution, and reviewable outcomes. The practical conclusion is that autonomy must be governed as a workflow, not as a prompt.

Guardrails define the identity and authority of the agent at runtime. Once an agent can act through tools, approvals, and delegated permissions, it begins to resemble a non-human identity with a scoped action surface. That is where IAM thinking becomes directly relevant: access should be task-bound, tool-bound, and revocable in the same way teams would constrain any other privileged workload. The practitioner takeaway is to manage agent authority as an identity control problem.

Eval failure often reveals a control-design failure upstream. A score that cannot see the live run context, the current policy boundary, or the actual user objective is not validating reliability, it is validating a narrow proxy. That creates a false sense of assurance for teams that think test scores equal production safety. The practitioner conclusion is to align eval design with the operational reality the agent will face.

For agentic systems, observability must feed remediation, not just reporting. A trace that shows what went wrong only matters if it changes the next run through a new test, a tighter permission boundary, or a revised tool policy. That makes feedback loops part of the security model, not just engineering hygiene. The practitioner conclusion is to connect failure evidence directly to guardrail updates and regression tests.

What this signals

Agent authority should be managed as a security boundary. When a system can choose tools, retry work, and branch into parallel actions, the practical control question becomes who or what is permitted to act on behalf of the user. That is the same governance problem IAM teams already solve for other delegated actors, and the difference is that the decision cycle is now machine-speed.

Evaluation without enforcement creates false confidence. A score can describe behaviour, but it cannot stop an agent from repeating the same bad path on the next run. Teams should therefore pair observability with hard constraints, because the reliability of agentic systems depends on both detection and prevention.

Longer autonomy windows increase the value of run evidence. The more steps an agent can take, the more important it becomes to reconstruct the trajectory, identify the failure point, and turn it into a regression test or permission change. In practice, that is how agent governance becomes operational rather than aspirational.

For practitioners

• Define task-scoped agent authority List which tools, actions, and approvals each agent may use for each workflow, and revoke anything not needed for the current task.
• Enforce code-level guardrails Convert high-risk behaviours such as sending messages, modifying files, or making external calls into executable restrictions rather than prompt wording.
• Instrument full run traces Capture tool calls, retries, branch decisions, and final outputs so reviewers can reconstruct how the agent reached a result.
• Design evals around production context Give judges the retrieved sources, policy rules, and user goal so scores reflect the real operating environment instead of stale assumptions.
• Feed failures back into the next build Turn recurring failures into regression tests, revised prompts, or tighter permissions so each run changes the system instead of just documenting the miss.

Key takeaways

• AI agent reliability depends on the harness as much as the model, because orchestration choices shape whether tool use stays controlled.
• Guardrails and evals are complementary controls, not substitutes, and both are needed as agents gain more time and authority.
• Teams that increase agent autonomy without tightening permissions, traces, and feedback loops will struggle to keep outcomes predictable.

Key terms

• Agent Harness: The agent harness is the runtime layer that wraps a model and turns it into an acting system. It usually includes the loop, tools, context handling, permissions, hooks, and logs. In security terms, it is often the real place where privilege sits and where identity evidence must be governed.
• Guardrails: Guardrails are policy controls that inspect prompts and model outputs against defined safety, privacy, and compliance rules. In AI operations, they reduce harmful language and disclosure risk, but they do not replace entitlement management, logging, or identity governance for the systems that call the model.
• Evaluations: Evaluations are structured checks used to judge whether an AI system produced the right result, not just fluent output. For coding agents, they often compare intent, root cause, and artifact quality against expected outcomes. Strong evals measure semantic correctness and workflow success across realistic inputs.
• Autonomy Window: The autonomy window is the amount of time or number of steps an AI agent can operate without human supervision. As that window expands, the agent’s reliability depends more on governance, authorization, and observability than on model quality alone.

Deepen your knowledge

Build the governance skills needed to control AI agents, machine identity, and access boundaries with the NHI Foundation Level course, the industry's only accredited NHI security programme. It is designed for practitioners who need to translate delegated authority into enforceable controls.