AI eval failures often start in product design, not the model

At a glance

What this is: This is an analysis of why AI evaluation failures often trace back to product design, missing context, and vague criteria rather than model quality alone.

Why it matters: It matters because teams building agentic and AI systems need evaluation processes that inspect traces, clarify intent, and separate product defects from model behaviour.

Context

AI evaluation only works when the system under test is well enough specified to make failure meaningful. If the product does not capture user intent, constraints, retrieval context, tool calls, or approval boundaries, a score can describe the output while hiding the real defect in the interaction design.

This article sits in the AI governance and agent engineering space rather than traditional IAM, but it still intersects with identity where tools, permissions, and account context shape what an agent is allowed to do. The core governance problem is not just whether the model is accurate, but whether the surrounding product and control plane let reviewers see the full decision path.

Key questions

Q: How should teams diagnose AI eval failures when the product design is weak?

A: Start by checking whether the system collected enough context, clarified ambiguous intent, and exposed the full trace. If those basics are missing, the eval is often measuring a product defect, not a model defect. Teams should separate prompt, retrieval, tool, permission, and criteria problems before they change the model.

Q: Why do AI eval scores change when the product is still evolving?

A: Because the scoring rule may be changing as teams learn what good behaviour actually looks like. New edge cases, revised user expectations, and clearer domain constraints can all shift the criteria without changing the model itself. Versioned criteria with ownership and change notes keep those changes auditable.

Q: What are the signs that trace review is missing the real failure?

A: If reviewers only see the final answer, they will miss the request ambiguity, the clarification path, the retrieved evidence, and the tool actions that shaped the result. That produces shallow labels like bad response instead of actionable diagnoses such as wrong tool call or missing approval boundary.

Q: How should security teams manage permissions for AI agents?

A: Security teams should regularly assess and update the permissions granted to AI agents to ensure they align with their intended scope. Implementing a governance framework that details access levels and usage policies is crucial to mitigate risks. Moreover, continuous monitoring can detect irregular permissions that may increase exposure.

Technical breakdown

Why ambiguous prompts distort AI eval results

An eval can only measure what the product actually asks the model to do. When a user request is underspecified, the model is forced to infer missing constraints, and the resulting output may look poor even though the upstream interface never resolved the ambiguity. In practice, this is a product design problem because the application should capture clarifying context before execution. That matters especially for agentic workflows where a single bad assumption can propagate into a tool call, a code change, or an external action.

Practical implication: collect structured intent, permissions, and constraints before consequential model or tool execution.

How criteria drift breaks AI evaluation programmes

Evaluation criteria are not static truths. As teams see real behaviour, they often discover that initial success definitions were too broad, too optimistic, or simply misaligned with how the product is used. That is criteria drift, where the yardstick changes as the system becomes better understood. Versioning criteria turns those changes into an auditable record instead of an invisible shift in judgement. Without that lineage, score movement can reflect changing standards as much as system improvement.

Practical implication: version evaluation criteria, owners, examples, and change reasons so score changes remain interpretable.

Why trace review beats metric-only diagnosis

Generic metrics such as correctness or helpfulness rarely reveal the system layer that caused a failure. Trace review shows the full chain, including the original request, clarification turns, retrieved evidence, tool calls, intermediate decisions, and final action. That makes it possible to map an observed failure to the real cause, whether that is a missing permission check, an incomplete prompt, or a flawed acceptance criterion. For AI security and governance teams, trace fidelity is the difference between symptom tracking and root-cause analysis.

Practical implication: preserve end-to-end traces so reviewers can tie failures to the exact control, prompt, or tool contract that broke.

NHI Mgmt Group analysis

Product design debt is becoming evaluation debt. When teams treat every weak agent output as a model defect, they hide the real governance problem in the product layer. AI systems that do not capture intent, context, and execution boundaries cannot produce meaningful eval signals. For practitioners, the right starting point is to make the system explainable enough that a failure has a clear owner.

Trace review is the control plane for AI quality. Metrics matter only after reviewers can see the request, clarification path, retrieved context, and tool activity that shaped the outcome. That is especially true when agents operate with permissions or account context that can change what the system is allowed to do. For practitioners, the eval workflow must expose the full decision path, not just the final answer.

Versioned criteria should be treated like governed policy, not ad hoc commentary. Once teams learn from real usage, success definitions will change, and that change needs lineage. This is a governance pattern with direct relevance to AI risk management because it distinguishes model drift, product drift, and review drift. For practitioners, the criterion itself needs ownership, examples, and change history.

AI governance now depends on the interface between identity, permissions, and evaluation. When an agent can call tools or operate under account context, reviewers need to know whether the system saw the right authorisation boundary before it acted. The field should stop treating permissioned execution as a separate topic from evals. For practitioners, trace review must include access context wherever an AI system can act on behalf of a user or service.

What this signals

Trace completeness is now an AI governance control. If a review system cannot show the original request, clarification path, retrieved evidence, and tool activity in one place, it cannot support reliable root-cause analysis. For teams running agents in production, that means evaluation design and observability design are the same conversation.

Criteria drift is inevitable, but unmanaged drift is a programme risk. Teams that discover new failure modes in production will revise what counts as acceptable behaviour. The governance task is to make that revision visible, owned, and versioned so quality metrics do not become moving targets.

When an AI system uses permissions or account context, identity becomes part of evaluation. The reviewer is no longer judging output alone but whether the system acted within the access boundary it was given. That makes trace review relevant to IAM, PAM, and agent governance programmes that need to know who or what was authorised to act.

For practitioners

• Instrument trace capture across the full agent journey Record the original request, clarification turns, retrieved context, tool calls, intermediate actions, final output, and version identifiers so reviewers can reconstruct each run.
• Require ambiguity resolution before consequential actions Use targeted follow-up questions, structured fields, or explicit confirmation when a request could map to multiple actions, especially where permissions or side effects differ.
• Version evaluation criteria as governed artefacts Assign owners, examples, and change rationales to each criterion so score movement can be traced to a model change, a product change, or a review change.
• Put domain experts on the first review pass Build an interface that surfaces request context, evidence, tool behaviour, and the criterion in one view so subject matter experts can label failures quickly and accurately.
• Map failures to the layer that actually broke Separate prompt defects, retrieval gaps, tool contract issues, permission mistakes, and genuine model errors before deciding whether to tune the model or redesign the product.

Key takeaways

• Many AI eval failures begin as product design failures because the system never collected enough context for the model to answer safely.
• Trace review is more useful than metric-only scoring because it exposes where the real fault sits in the request, tool, permission, or criteria path.
• Versioned criteria and domain expert review turn AI evaluation from a blunt scorecard into a governed diagnostic process.

Key terms

• Trace Review: The process of examining the full sequence of an AI system's behaviour, including the request, context, tool calls, intermediate steps, and final output. It is the most reliable way to identify whether a failure came from the model, the product, or the operating conditions around it.
• Classification Drift: Classification drift is the gradual mismatch between a system's labels and the real sensitivity of the content as files change over time. It happens when documents are edited, copied, or repurposed faster than the model or rules are updated, creating gaps between visibility and actual protection.
• Ambiguity Resolution: Ambiguity resolution is the process of disambiguating a user request before an AI system takes action. It may involve asking follow-up questions, collecting structured fields, or confirming constraints so the model does not guess at intent with real-world side effects.
• Identity Lineage: Identity lineage is the traceable relationship between a human owner and the non-human identities that person creates, authorises, or depends on. It allows security teams to connect service accounts, API keys, tokens, and AI agents back to accountable ownership for review, audit, and retirement decisions.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps practitioners connect identity controls to the broader governance demands of modern AI systems.