AI agent production failures are usually context and eval gaps

At a glance

What this is: Arize argues that AI agents usually fail in production because teams underinvest in context, retrieval and evaluation, not because the model itself is the only problem.

Why it matters: IAM and AI security teams need to treat agent reliability as a governance issue, because grounded context, traceability and evaluation determine whether agentic systems can be trusted to act safely.

Context

AI agents often appear stable in demos because the inputs are curated, the scope is narrow and the workflow is tightly controlled. Production changes the equation: data is messier, systems are older, and the agent must make decisions under uncertainty while interacting with tools and business processes.

The governance gap is not just model quality but operational context, evaluation and traceability. For AI agents, that becomes an identity and access question as well, because the value of an agent depends on what data it can see, what tools it can invoke and how its actions are reviewed.

Key questions

Q: What breaks when AI SOC agents do not have enough context?

A: They become brittle, overconfident, and inconsistent because they can only act on the visible event, not the organisational reasoning behind it. That leads to false negatives, weak attribution, and repeated investigations that humans would resolve faster. Context is not optional metadata in SOC automation. It is the basis for trustworthy machine judgement.

Q: Why do AI agent evals matter more than benchmark scores?

A: Benchmarks show what a model can do in a controlled test, but evals show whether the full agent can succeed in the real workflow. A production agent needs task success, correct tool calls, edge-case handling and consistency, all measured against scenarios that reflect the actual operating environment. Without that, teams cannot tell if a change improved reliability or just changed the output style.

Q: How do security teams reduce AI agent mistakes without over-restricting them?

A: Use scoped retrieval, explicit tool authorisation and traceable decision paths so the agent can act within a defined boundary. That preserves useful automation while making failures observable and recoverable. The aim is not to block agent behaviour, but to ensure every action is tied to approved data, permissions and workflow context.

Q: Should organisations prioritise observability or model upgrades for AI agents?

A: Observability should come first because it tells you whether the agent is actually failing, where it failed and what changed. Model upgrades without tracing and evals often increase cost without improving outcome quality. Once the system is measurable, model changes become safer and easier to justify.

Technical breakdown

Why context, not model size, determines agent reliability

In production, an AI agent is only as reliable as the information boundaries around it. Context includes system data, retrieval results, memory, workflow state and domain constraints, while the model itself is the reasoning layer that interprets that context. When teams swap in a larger model without improving the surrounding data and tool context, they often get better language output but no better decisions. That is why agents can sound confident and still answer the wrong question. The failure is structural: the agent is reasoning without enough grounded state to distinguish a plausible response from the correct one.

Practical implication: Treat context as a control surface, not a prompt-writing detail, and govern what information the agent can use before it reasons.

Retrieval and reasoning should be separated in agent architecture

A robust agent design separates fact gathering from decision making. Retrieval pipelines fetch current records, documents or telemetry from source systems, then the model evaluates that grounded data instead of inventing facts from memory. This reduces hallucination risk and makes the agent’s behaviour more auditable because the inputs are explicit. It also creates a cleaner boundary for access governance: the retrieval layer controls what the agent can know, and the reasoning layer controls how it acts on that knowledge. For AI agents, that boundary is the identity and authorisation layer in practice, even when teams do not label it that way.

Practical implication: Give agents scoped retrieval paths and explicit tool permissions so access and reasoning remain separable and reviewable.

Evals turn agent behaviour into something teams can measure

Evaluation is what converts an impressive demo into a system you can trust over time. Good evals test task success, tool-call correctness, consistency across variations and behaviour under edge cases, ideally against golden datasets built from real scenarios. Without that loop, teams discover failures only after users or operations are affected. For production agents, observability and evals are governance controls because they reveal whether the system is drifting, overconfident or misusing its tool access. They also create evidence for change management when models, prompts or retrieval logic change.

Practical implication: Build repeatable evals before scale-up so agent changes are measured against stable operational outcomes, not intuition.

NHI Mgmt Group analysis

Agent reliability is a governance problem before it is a model problem. The article’s core lesson is that production failures usually come from missing context, weak retrieval boundaries and absent evaluation, not from a lack of model sophistication. That shifts the control point from model selection to system design, where identity, access and traceability determine what the agent can safely do. For practitioners, the right question is not whether the model is smarter, but whether the surrounding controls make its behaviour defensible.

Context loss is the named failure mode behind many agent breakdowns. When an agent does not have the right operational state, it can still produce fluent output that is confidently wrong. That is especially dangerous in environments where tool use and data access carry real business consequences, because the failure is not obvious until an action is taken. The practical lesson is that context completeness must be treated as a control objective, not a convenience layer.

Traceability is the missing bridge between AI output and operational accountability. The article shows that teams need to know what data the agent retrieved, which tools it called and why a decision was made. That is the same governance logic used in IAM and PAM, where access must be attributable to a defined identity and scope. For practitioners, the implication is clear: if you cannot explain an agent’s decision path, you cannot safely delegate production work to it.

Agentic AI security now intersects directly with access governance. AI agents are not just software consumers of data, they are runtime actors whose permissions determine operational risk. That makes scoped retrieval, tool authorisation and evaluation part of the identity programme, not an adjacent engineering concern. The article reinforces that agent governance fails when teams treat action-taking systems as if they were only text generators.

Retrieval-backed context: the most useful control pattern is to make the agent reason over current, governed data rather than cached assumptions or prompt memory. This reduces silent drift and makes access decisions more reviewable across the agent lifecycle. For practitioners, the conclusion is to govern the data path first, then judge the model’s reasoning on top of it.

From our research library:

• 62% of organisations are at least experimenting with AI agents, but only 23% have scaled one into a single business function, according to McKinsey's 2025 State of AI survey.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Context debt is the hidden failure mode in production agents: teams often improve model capability faster than they improve the data, retrieval and workflow boundaries the agent depends on. That creates a reliability gap that will keep widening as agents move from prototypes into operational systems.

At scale, agent governance will depend less on whether a model can answer and more on whether the surrounding system can prove what it saw, what it did and why it acted. That pushes evaluation, tracing and scoped tool access into the centre of AI operations, not the margin.

70% of organisations grant AI systems more access than they would give a human employee performing the exact same job, according to the 2026 Infrastructure Identity Survey. For practitioners, that gap means production controls must start at issuance time, not after the agent is already acting.

For practitioners

• Implement retrieval-backed agent workflows Feed the agent current records, documents or telemetry from controlled source systems instead of relying on prompts or model memory. Use explicit retrieval boundaries so the agent only reasons over data it is authorised to see.
• Define and review tool permissions for each agent Map every tool the agent can call, then scope those permissions to the smallest workflow needed for the task. Revisit those entitlements whenever the workflow changes or the agent is repurposed.
• Build golden datasets from real cases Create a compact evaluation set from actual production scenarios, edge cases and expected outcomes. Use it to compare model changes, prompt changes and retrieval changes before deployment.
• Add tracing to every production agent flow Record retrieved context, tool calls and final outputs so investigators can reconstruct the agent’s decision path. Use those traces to detect drift, diagnose errors and support post-change review.

Key takeaways

• AI agent failures in production usually stem from missing context, weak retrieval boundaries and inadequate evaluation rather than from model quality alone.
• The operational risk rises when agents are allowed to act on ungoverned data or opaque tool permissions, because confidence is not the same as correctness.
• Teams that can trace, test and scope agent behaviour will have a far better path to reliable deployment than teams chasing the newest model.

Key terms

• Agent Context Protocol: An agent context protocol is a communication layer that lets AI agents exchange structured context, permissions, and task data with other systems or agents. It creates a shared operational surface for memory and tool use, which makes access control, authentication, logging, and context validation critical to safe deployment.
• Golden Dataset: A curated set of expected inputs and outputs used to test whether an AI system behaves consistently across changes. In governance terms, it becomes evidence that model or workflow changes did not introduce regressions that would affect reliability, safety, or access-related behaviour.
• Agent Tracing: Agent tracing records the steps an AI agent takes while reasoning, calling tools, retrying actions, and producing outputs. It helps teams reconstruct why an agent behaved a certain way, where latency was introduced, and whether the workflow stayed within approved boundaries.
• Retrieval-Backed Workflow: An agent design pattern that pulls current facts from governed source systems before the model reasons over them. It reduces dependence on model memory and improves reliability by separating data retrieval from decision making.

Deepen your knowledge

NHI Mgmt Group's NHI Foundation Level course covers NHI governance, machine identity security, and agentic AI identity, the industry's only accredited NHI security programme. It helps security, identity and platform teams build the governance foundations needed for safe production automation.