Notifications
Clear all
Topic starter
19/08/2026 3:28 pm
TL;DR: Production AI agents fall into three patterns, customer-facing, internal enterprise, and developer platform, and each demands different harnesses, eval plans, rollout controls, and failure management, according to Arize. The key lesson is that shipping agents is less about model choice than about governing context, permissions, observability, and bounded execution.
NHIMG editorial — based on content published by Arize: 3 production patterns for AI agents and how to evaluate each one
Questions worth separating out
Q: How should security teams govern agentic AI as it moves into production?
A: Security teams should govern agentic AI as a class of non-human identity, not as a generic application feature.
Q: Why do AI agents create new risk in non-human identity management?
A: AI agents create risk because they operate as software identities with delegated authority, but many organisations do not track them with the same discipline applied to users or service accounts.
Q: What breaks when teams rely on notebook-style evaluation for production agents?
A: Notebook-style checks usually test final answers, not the sequence of tool calls, retrieved context, and workflow decisions that actually determine safety.
Practitioner guidance
- Standardise the agent harness Define one reusable harness pattern for tools, permissions, sandboxes, and workflow controls before more agents are approved for production.
- Evaluate traces at the level of real production tasks Score full traces for tool use, context retrieval, and workflow steps, then tie each failure mode to a binary or rubric-based evaluator.
- Separate customer, internal, and platform governance Assign different rollout gates, observability thresholds, and approval paths to customer-facing, internal enterprise, and developer platform agents.
What's in the full article
Arize's full article covers the operational detail this post intentionally leaves for the source:
- Pattern-by-pattern rollout guidance for customer-facing, internal enterprise, and developer platform agents
- The specific trace, eval, and observability layers used to detect production failure modes
- Practical examples of harness design, including tools, permissions, sandboxes, and approval flow choices
- How the authors distinguish session, trace, and span evaluation for different deployment goals
👉 Read Arize's analysis of production AI agent patterns and evaluation →
Production AI agents: what pattern are you actually shipping?
Explore further
19/08/2026 4:12 pm
Production AI agents are becoming non-human identities with operational reach. Once an agent can call tools, access files, or trigger workflows, it is no longer just an application feature. It becomes a runtime actor whose privileges, session scope, and audit trail need identity-style governance. That is why agent programmes should be reviewed through the lens of bounded access and verifiable action, not only model quality. Practitioners should classify agents as governed actors before they are allowed to interact with production systems.
A question worth separating out:
Q: How should organisations decide whether to expand or pause an agent rollout?
A: Use production traces, not enthusiasm, as the release signal. If cost spikes, tool misuse, or context gaps appear in a small rollout, stop and fix the harness before increasing scope. Expansion should follow observed control stability across real sessions, not just a successful demo or a clean internal test set.
👉 Read our full editorial: Production AI agent patterns expose new governance and eval gaps
