Notifications
Clear all
Topic starter
20/08/2026 5:50 pm
TL;DR: AI agent demos fail in production when teams over-focus on model upgrades and underinvest in context, retrieval, and evaluation, according to Arize’s April 23, 2026 analysis of agent engineering. The governance lesson is that reliable agent behaviour depends on grounded data, measurable outcomes, and traceable decision paths, not raw model capability.
NHIMG editorial — based on content published by Arize: Beyond models, how context and evals make agents work in production
Questions worth separating out
Q: How should security teams govern AI agents that can access enterprise systems?
A: Security teams should govern AI agents as non-human identities with explicit ownership, scoped privileges, and continuous monitoring.
Q: Why do AI agents that succeed in demos fail so often in production?
A: Demos hide the conditions that break real workflows, including expired tokens, changing APIs, concurrent activity, and incomplete context.
Q: What do security teams get wrong about governing AI agents?
A: They often treat agents like another automation layer instead of governed non-human actors with their own access paths.
Practitioner guidance
- Implement retrieval-backed agent workflows Replace prompt-only flows with retrieval from authoritative systems of record, and require the agent to ground every consequential action in cited operational data rather than inferred memory.
- Create golden datasets from real workflows Build a small but representative evaluation set with domain experts, then score task success, tool-call correctness, and edge-case behaviour before allowing production changes.
- Separate reasoning from data retrieval Design the architecture so search, context assembly, and reasoning are observable as distinct steps, which makes it easier to spot whether failure came from bad data, bad policy, or bad interpretation.
What's in the full article
Arize's full blog post covers the operational detail this post intentionally leaves for the source:
- A deeper walkthrough of the agent engineering workflow used to separate retrieval, reasoning, and tool execution.
- Practical examples of golden dataset creation for domain-specific evaluation in production agent systems.
- The article's discussion of tracing, observability, and how teams investigate failed agent runs at scale.
- Additional commentary on building versus buying AI observability infrastructure for agent workflows.
👉 Read Arize’s analysis of why context and evals determine AI agent production reliability →
AI agent reliability in production: what context and evals fix?
Explore further
20/08/2026 6:29 pm
Production AI agents create a context trust gap, not just a model quality problem. The article is right to shift attention away from model selection and toward the systems that supply facts, state, and constraints. In agentic environments, the governance issue is whether the agent can act on trustworthy context before it takes tool actions. That intersects directly with identity and NHI governance because every delegated action depends on trusted access paths, traceability, and policy enforcement. Practitioners should treat context as part of the control plane, not as an implementation detail.
A question worth separating out:
Q: How do security teams know whether an AI agent is operating safely?
A: Security teams know an AI agent is operating safely when its permissions, invoked tools, and accessed data remain consistent with the approved use case over time. Useful signals include restricted data exposure, unchanged guardrails, and a stable identity path. If any of those drift, the agent should be re-reviewed before it expands further.
👉 Read our full editorial: AI agent production failures are usually context and eval gaps
