Production AI agent patterns expose new governance and eval gaps

At a glance

What this is: This article maps three common production AI agent patterns and shows that deployment risk is determined by harness, eval, and rollout design rather than model class alone.

Why it matters: For IAM, PAM, and NHI practitioners, the article matters because production agents introduce tool access, permissions, and runtime behaviour that must be governed like other high-risk identities and workflows.

👉 Read Arize's analysis of production AI agent patterns and evaluation

Context

Production AI agents create a governance gap because teams often treat them as model deployments when they behave more like software entities with tools, permissions, memory, and runtime action. That changes the control problem from prompt quality to who or what can act, on which systems, under what constraints, and with what auditability. In identity terms, agents can behave like non-human identities that need bounded access and traceable privileges.

Arize’s framing is useful because it separates agent pattern from implementation detail. A customer-facing assistant, an internal enterprise workflow agent, and a developer platform agent all create different risk profiles, but each still depends on controlled credentials, reliable observability, and disciplined rollout. That makes the overlap with IAM, NHI governance, and PAM explicit rather than incidental.

For practitioners, the starting position described here is typical of early agent programmes: teams build around model capability first and governance later. That sequencing is where many production failures begin.

Key questions

Q: How should security teams govern agentic AI as it moves into production?

A: Security teams should govern agentic AI as a class of non-human identity, not as a generic application feature. That means assigning ownership, scoping permissions tightly, logging every tool action, and revoking access on a defined lifecycle. Production rollout should require clear approval points for high-risk actions and continuous monitoring for drift.

Q: Why do AI agents create new risk in non-human identity management?

A: AI agents create risk because they operate as software identities with delegated authority, but many organisations do not track them with the same discipline applied to users or service accounts. They can connect quickly, persist across teams, and accumulate permissions that are hard to review. That combination increases the chance of unnoticed access drift and credential exposure.

Q: What breaks when teams rely on notebook-style evaluation for production agents?

A: Notebook-style checks usually test final answers, not the sequence of tool calls, retrieved context, and workflow decisions that actually determine safety. That means a run can look correct while still touching the wrong data or taking the wrong operational path. Production evaluation needs traces, failure modes, and repeatable scoring tied to real sessions.

Q: How should organisations decide whether to expand or pause an agent rollout?

A: Use production traces, not enthusiasm, as the release signal. If cost spikes, tool misuse, or context gaps appear in a small rollout, stop and fix the harness before increasing scope. Expansion should follow observed control stability across real sessions, not just a successful demo or a clean internal test set.

Technical breakdown

Production agent patterns and why they fail differently

Arize’s three-pattern split matters because the operational boundary is not the model, it is the execution context. Customer-facing agents depend on account-scoped data and external users, so cost and context quality become primary risks. Internal enterprise agents depend on fragmented systems and multi-step workflows, which creates tool sprawl and governance friction. Developer platform agents centralise primitives such as auth, tracing, and deployment, so they fail when teams cannot standardise those controls across projects. The technical point is that each pattern needs different harness assumptions, evaluation coverage, and permission boundaries, even if the underlying model is identical.

Practical implication: Treat the production pattern as a control-design decision before you ship any agent.

Why agent harnesses matter more than prompt quality

A harness is the runtime envelope around an agent: tools, permissions, sandboxes, branching logic, and guardrails that keep behaviour bounded. In practice, this is where governance lives. If the harness is weak, the agent can take the wrong action even when the model answer looks reasonable. That is especially relevant for tool-using systems because access to APIs, files, ticketing, and internal data turns a language model into an operational actor. Evaluation should therefore inspect traces, not just final outputs, and the harness should make failure modes visible enough to debug and restrict.

Practical implication: Define tool permissions and sandbox boundaries as part of the build, not as a post-launch patch.

Evaluations must track traces, not just outputs

Agent evaluation is different from traditional software testing because the correct path is often under-specified. An agent may call different tools or choose a different sequence and still complete the task safely. That is why trace-level scoring is essential: it captures tool calls, workflow steps, context use, and final outcomes. Observability tells you what happened; evals tell you whether it was acceptable. For production teams, the useful unit is usually a span, trace, or session, depending on whether the goal is debugging, regression control, or user-level cost and safety monitoring.

Practical implication: Build evaluators around trace evidence and failure modes, not around a single expected call sequence.

Threat narrative

Attacker objective: The attacker objective is to make the agent execute unintended actions, misuse privileges, or expose sensitive operational context at production scale.

1. Entry occurs when a production agent is given broad tool access or external user input without a sufficiently bounded harness. Escalation happens when the agent can chain retrieval, tools, and workflow steps into actions the operator did not intend. Impact follows when those actions touch sensitive systems, privileged data, or costly downstream operations without clear guardrails.

NHI Mgmt Group analysis

Production AI agents are becoming non-human identities with operational reach. Once an agent can call tools, access files, or trigger workflows, it is no longer just an application feature. It becomes a runtime actor whose privileges, session scope, and audit trail need identity-style governance. That is why agent programmes should be reviewed through the lens of bounded access and verifiable action, not only model quality. Practitioners should classify agents as governed actors before they are allowed to interact with production systems.

Harness design is the control plane that most teams still underestimate. The article correctly distinguishes primitives from product features, but the security meaning is deeper: harnesses are where permissions, sandboxes, and workflow constraints are enforced. If those controls are improvised per team, every agent becomes a local exception. That leads to governance debt, because the organisation inherits dozens of slightly different trust boundaries with no consistent review path. Practitioners should standardise the harness before standardising the agent.

Trace-based evaluation is now an access control problem as much as a quality problem. The same trace that proves an answer was reasonable also proves whether the agent touched the right data, invoked the right tools, and stayed inside policy. That makes observability and evaluation inseparable from auditability. In identity terms, the organisation needs evidence of what the agent could do and what it actually did. Practitioners should treat traces as governance records, not just debugging artefacts.

Context engineering creates the new blast radius. Customer-facing and internal enterprise agents both depend on the quality of retrieved context, but the failure mode is governance drift rather than model failure. The more systems an agent can reach, the more permissions and data pathways must be explicitly constrained. This is where OWASP Agentic AI Top 10 and NIST AI Risk Management Framework thinking align with identity controls. Practitioners should assume that uncontrolled context access expands the attack surface faster than model updates shrink it.

Named concept: agent governance debt. When teams ship multiple agent patterns without a shared harness, evaluation model, or permission architecture, they accumulate hidden control variation that becomes expensive to unwind. The result is not just technical inconsistency but audit uncertainty and slower incident response. That debt compounds as agents move from pilot to production. Practitioners should reduce it by making one governed pattern reusable across programmes.

What this signals

Agent governance debt: many programmes will discover that their first production issue is not model quality but inconsistent control design across agent types. That means IAM and PAM teams should get involved before rollout, because tool permissions, retrieval scope, and workflow approvals are now part of the identity boundary. Referencing the OWASP Top 10 for Agentic Applications 2026 alongside the NIST AI Risk Management Framework helps anchor that boundary in recognised governance language.

Production teams should expect traces to become governance evidence, not just diagnostics. Once an agent can operate across multiple systems, the audit question changes from whether the model answered correctly to whether it stayed within approved action paths. That pushes identity, application, and security teams toward shared review of permissions, observability, and exception handling.

The practical signal for the next 12 months is that agent programmes will split into governed patterns rather than generic deployments. Organisations that standardise harnesses and evaluation early will be able to scale faster because they can reuse controls, not just code. Those that cannot will keep rediscovering the same failure modes under different project names.

For practitioners

• Standardise the agent harness Define one reusable harness pattern for tools, permissions, sandboxes, and workflow controls before more agents are approved for production. Make exceptions explicit and time-bound so they can be reviewed against the same policy baseline.
• Evaluate traces at the level of real production tasks Score full traces for tool use, context retrieval, and workflow steps, then tie each failure mode to a binary or rubric-based evaluator. Use the output to decide whether to tighten permissions, change retrieval scope, or revise approval logic.
• Separate customer, internal, and platform governance Assign different rollout gates, observability thresholds, and approval paths to customer-facing, internal enterprise, and developer platform agents. The same control set will not fit all three patterns, especially where user exposure and privilege differ.
• Treat agent credentials like governed non-human identities Inventory the credentials, tokens, and service permissions each agent can use, then enforce least privilege and review ownership before production rollout. Reassess access whenever tools, data sources, or workflow scope changes.
• Run staged release with failure-driven expansion Release to a small user or workflow slice first, then expand only when the trace data proves the harness, cost model, and evaluation coverage are holding. Build new checks from observed failures rather than from assumptions.

Key takeaways

• Production AI agents need governance by pattern, not just by model choice.
• Trace-level evaluation is the clearest way to see whether an agent stayed inside its intended boundary.
• Teams that standardise harnesses and permissions early will have a far easier path to scaling agent programmes safely.

Key terms

• Production Agent Pattern: A production agent pattern is the operating shape an AI agent takes in production, such as customer-facing, internal enterprise, or developer platform. The pattern determines the harness, rollout model, evaluation scope, and risk profile, even when the same foundation model is used underneath.
• Agent Harness: The agent harness is the runtime layer that wraps a model and turns it into an acting system. It usually includes the loop, tools, context handling, permissions, hooks, and logs. In security terms, it is often the real place where privilege sits and where identity evidence must be governed.
• Trace-Based Evaluation: An evaluation approach that records the full execution path of a run, including inputs, intermediate calls, retrieved context, and outputs. It helps teams debug multi-step AI systems by showing how a result was produced, not just whether it looked correct.
• Agent Governance: Agent governance is the set of policies, controls, and evidence required to manage autonomous software as a non-human identity. It covers consent, tool access, lifecycle review, audit logging, and revocation so that an agent remains bounded as its workflows change.

What's in the full article

Arize's full article covers the operational detail this post intentionally leaves for the source:

• Pattern-by-pattern rollout guidance for customer-facing, internal enterprise, and developer platform agents
• The specific trace, eval, and observability layers used to detect production failure modes
• Practical examples of harness design, including tools, permissions, sandboxes, and approval flow choices
• How the authors distinguish session, trace, and span evaluation for different deployment goals

👉 Arize's full post covers the pattern table, eval loop, and rollout checklist in more operational detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and agentic AI identity. It is designed for practitioners who need to align identity controls with modern automated systems and production access paths.