AI eval failures often start in product design, not the model

At a glance

What this is: The article argues that many AI eval failures come from product design, not model quality, because teams score outputs before they understand the system decisions that produced them.

Why it matters: This matters to IAM practitioners because agentic systems increasingly make access-adjacent decisions, and weak context handling or unclear criteria can turn governance gaps into false confidence about control effectiveness.

👉 Read Arize's analysis of why AI evals fail before they begin

Context

AI evaluation is only useful when the system under test is clearly specified. If an agent receives ambiguous instructions, missing context, or poorly defined success criteria, the resulting score can reflect interface design failure rather than model performance. That distinction matters for AI governance, because teams that supervise agentic systems need to know which layer actually failed before they can assign control ownership.

For identity and access programmes, the article’s point maps cleanly to delegated decision-making. When an AI system can propose actions, call tools, or interpret user intent, it starts to resemble a governed runtime component rather than a simple application feature. In that setting, evals need to expose whether the issue is identity context, permissions, tool contracts, or model behaviour, not just the final answer.

Key questions

Q: How should teams evaluate agentic AI systems without confusing product failures with model failures?

A: Start with the trace, not the score. Review the request, context assembly, tool calls, intermediate decisions, and final action so you can determine whether the failure came from product design, missing context, permissions, or the model itself. If reviewers cannot see the full trajectory, the evaluation will overstate model blame and understate system design problems.

Q: Why do AI eval criteria change after teams see the system in production?

A: Because generative and agentic products reveal requirements that were not obvious in design-time specifications. A response can be accurate yet still violate policy, create an unauditable result, or exceed acceptable scope. Teams should expect criteria drift and manage it deliberately through versioning, ownership, and clear examples rather than treating it as an eval defect.

Q: What do security teams get wrong about governing AI agents?

A: They often treat agents like another automation layer instead of governed non-human actors with their own access paths. Once an agent can connect to tools and data at runtime, the programme needs attribution, scoped privileges, and lifecycle oversight. Otherwise, the agent becomes an unreviewed extension of the enterprise access model.

Q: How can organisations make AI trace review useful for governance and accountability?

A: Put the relevant evidence in one place: request, clarification, retrieved context, tool activity, outcome, and the criterion being assessed. Then let subject-matter experts label the failure quickly and explain why it matters. That turns trace review into a repeatable governance process instead of a slow forensic exercise.

Technical breakdown

Why ambiguous requests break AI evals

A model can only evaluate or act on the information it receives. If the application forwards a vague request without clarifying scope, constraints, or approval boundaries, the model is forced to guess, and the evaluator later misattributes the failure to the model. In agentic systems, this is often a product design issue: the interface should collect structured context before any tool call, especially where the action can affect data, code, or access. Trace review only becomes meaningful when reviewers can see both the ambiguity and the system’s attempt to resolve it.

Practical implication: capture clarification turns and required fields before execution so evals can separate user ambiguity from model error.

How criteria drift changes what success means

Evaluation criteria are not static when a product is still evolving. Teams often discover, only after seeing real behaviour, that a response can be technically correct yet operationally unacceptable because it breaks policy, omits disclosure, or creates an unauditable result. That is why criteria should be treated as versioned product artifacts with owners, examples, and change history. Without that discipline, a score change could reflect shifting expectations rather than actual quality improvement, making governance reports difficult to trust.

Practical implication: version criteria like code, and record why each rule changed so score movement remains interpretable.

Why trace review beats generic metrics

Broad metrics such as helpfulness or groundedness are too coarse to explain many failures. A trace gives the sequence that matters: request, clarification, retrieved context, tool calls, intermediate decisions, final action, and product version. That full trajectory lets reviewers identify the exact system layer that caused the failure, whether it was context assembly, permissions, retrieval, or prompt behaviour. For identity-linked agent workflows, the same logic applies to access decisions: the problem may be missing account context or an over-broad tool contract, not the final output alone.

Practical implication: review complete traces before tuning metrics so you can fix the layer that actually produced the bad result.

NHI Mgmt Group analysis

AI evaluation is a governance control, not just a measurement exercise. When teams score outputs without validating the system conditions that produced them, they mistake symptoms for root cause. That creates false assurance in agentic workflows where permissions, tool contracts, and context assembly all shape the outcome. The right frame is control effectiveness, not model beauty contest, because the control problem sits across product design, access, and runtime behaviour.

Query disambiguation is a hidden identity and authorisation problem. When an agent asks for or infers intent, it is effectively deciding whether it has enough context to act safely. In identity-linked systems, that means the request boundary, user context, and allowed action scope must be explicit before execution. Otherwise, the evaluator is judging a system that was never given a valid authorisation context in the first place.

Criteria drift creates evaluation debt: the longer teams keep changing success rules informally, the harder it becomes to know whether the system improved or the standard moved. That debt is especially risky in regulated or high-impact workflows where an auditable rationale matters as much as the answer. Teams should treat criteria versioning as part of AI governance, not an administrative extra.

Trace-based diagnosis should become the default operating model for agent review. The article’s strongest lesson is that expert judgment scales only when reviewers can see the full decision path, not just the endpoint. In practice, that aligns with AI governance, NIST AI RMF-style accountability, and access-control thinking for agentic systems. Practitioners should design for explainable review before they chase another dashboard.

What this signals

AI teams are moving toward trace-first governance, and identity programmes should expect the same standard to spread into agent permissions and delegated actions. The practical shift is simple: if an AI system can request context, invoke tools, or act on behalf of a user, the review model must prove that the action was authorised and explainable before the output is trusted.

The strongest control pattern here is not another generic metric but a better evidence chain. For identity and access teams, that means linking the agent's request, its context, the permissions it used, and the approval path into one reviewable record. That is where governance becomes operational rather than performative.

For practitioners

• Instrument full agent traces Capture the original request, clarification turns, retrieved context, tool calls, intermediate decisions, and final action so reviewers can identify the failing layer, not just the failing output.
• Version eval criteria as governed artifacts Assign an owner, change rationale, examples of acceptable and unacceptable behaviour, and review date to each criterion so score movement stays interpretable across releases.
• Add clarification gates before consequential actions Require structured fields or follow-up questions when a request could affect code, data, approvals, or access, and log the clarified intent alongside the original request.
• Put domain experts on the fastest review path Design a single review view that shows request, context, tool behaviour, and criterion together, because the fastest route to better evals is reducing the time it takes an expert to diagnose a trace.

Key takeaways

• Many AI eval failures are product design failures in disguise, because ambiguous requests and missing context distort the score before the model is fairly tested.
• Trace review is the practical way to separate model issues from context, permission, and workflow problems, especially in agentic systems.
• Governance improves when eval criteria are versioned, domain experts can review quickly, and every consequential action has a clear evidence trail.

Key terms

• Evaluation Criteria Drift: The gradual change in what a team considers a successful AI output as the product, users, or reviewers learn more from real-world behaviour. In practice, it is a governance problem as much as a model problem, because the scoring rule itself changes over time and can blur accountability.
• Trace Review: The process of examining the full sequence of an AI system's behaviour, including the request, context, tool calls, intermediate steps, and final output. It is the most reliable way to identify whether a failure came from the model, the product, or the operating conditions around it.
• Query Disambiguation: The act of resolving an ambiguous user request before an AI system takes action. It matters because unresolved ambiguity forces the model to guess, which can produce technically plausible but operationally unsafe outcomes, especially when the request affects code, data, or access.

What's in the full article

Arize's full article covers the operational detail this post intentionally leaves for the source:

• Examples of trace fields and review layouts that make agent error analysis faster for domain experts
• The specific eval artefacts Husain recommends versioning, including criteria ownership and change history
• How teams distinguish product issues from model issues when ambiguous requests produce poor outcomes
• The practical review loop Arize describes for turning repeated trace failures into scoped eval cases

👉 Arize's full article covers the trace-review workflow, criteria drift examples, and the product failures that look like model errors.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It gives security and identity practitioners a practical way to govern delegated systems and the access paths they create.