Notifications
Clear all
Topic starter
18/08/2026 2:19 pm
TL;DR: Agent reliability now depends less on model choice than on the context layer and harness engineering around retrieval, tools, permissions, memory, and handoffs, with evals needed to expose failures that benchmarks miss, according to Arize. That shift matters because enterprises are effectively governing AI system behaviour, not just model output, and stale or conflicting context can create confident irrelevance.
NHIMG editorial — based on content published by Arize: You chose the best model. Why is your agent still failing?
By the numbers:
- Arize says customers run more than 100 million evals every month across their AI systems.
- When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.
Questions worth separating out
Q: What breaks when AI systems reuse stale context after an error?
A: The main failure is loss of conversation isolation.
Q: Why do AI agents increase non-human identity risk?
A: AI agents increase non-human identity risk because they can execute many actions quickly once they inherit a credential or tool permission.
Q: How do you know if agent evaluation is actually working?
A: It is working when it explains failures well enough to change the system, not just when it outputs a score.
Practitioner guidance
- Implement trace-level evaluation Instrument every agent path so teams can see retrieval, tool calls, handoffs, and execution outcomes.
- Govern context as a versioned asset Assign owners, versioning, approval steps, and deprecation signals to business definitions, trusted sources, and policies.
- Scope agent permissions to task boundaries Limit tools, memory, and code execution to the minimum required for the specific workflow, then verify those boundaries in production.
What's in the full article
Arize's full analysis covers the operational detail this post intentionally leaves for the source:
- Trace examples showing how trajectory failures emerge across retrieval, tool use, memory, and handoffs
- The evaluator design choices behind code checks, LLM judges, and agent judges
- The operating loop for turning failed runs into regression data and engineering actions
- How the context layer is mined, versioned, approved, and activated across enterprise systems
👉 Read Arize's analysis of why enterprise AI agents still fail despite better models →
Agent failures despite good models: what context and evals fix?
Explore further
18/08/2026 3:11 pm
Confident irrelevance is the right name for a growing enterprise AI failure mode. The article shows that a capable model can still act on stale or conflicting business context and produce a fluent but wrong outcome. That is not a hallucination problem alone, it is a control problem caused by weak context governance and incomplete evaluation. Teams should treat this as a governance debt issue, because the risk grows every time the business changes faster than the context layer.
A question worth separating out:
Q: Should organisations treat context engineering as a governance control?
A: Yes. Context engineering governs what an agent is allowed to know, which sources it trusts, and which business definitions it uses before acting. That makes it a control surface for AI governance, data governance, and identity governance together. Without ownership, versioning, and approval paths, context drift becomes a security and reliability risk.
👉 Read our full editorial: Context and evaluation now define reliable enterprise AI agents
