Who should decide what good looks like in AI evaluation?

PMs and domain experts should define the quality standard because they understand user expectations, policy constraints, and business tradeoffs. Engineers can build the harness and the labeling workflow, but they should not be the only people deciding which failures matter or when a system is safe to ship.

Why This Matters for Security Teams

“Good” in ai evaluation is not a technical preference, it is a governance decision about acceptable risk, user impact, and accountability. If engineers define success alone, the evaluation can drift toward measurable system outputs while missing harms that matter to the business, the customer, or the regulator. NIST’s NIST Cybersecurity Framework 2.0 is useful here because it treats governance as a first-class security function, not an afterthought.

For AI systems, that means evaluation criteria should reflect policy constraints, model limitations, and the operational context in which outputs will be used. A chatbot serving internal support, a clinical triage assistant, and a fraud-detection model cannot share the same definition of acceptable performance. Domain experts understand the edge cases, legal constraints, and user expectations that determine whether a failure is merely inconvenient or genuinely unsafe. PMs bring product intent and risk tolerance into the decision, while security, legal, and compliance stakeholders ensure the evaluation does not ignore misuse, leakage, or unsafe automation.

In practice, many security teams encounter weak AI evaluation only after an apparently successful pilot has already produced unsafe decisions, policy violations, or unreviewed exceptions.

How It Works in Practice

Deciding what good looks like usually starts with a cross-functional rubric. The best practice is evolving, but current guidance suggests the rubric should define target behaviors, disallowed behaviors, escalation thresholds, and the evidence needed to approve release. For AI systems with external side effects, that rubric should also distinguish between model quality, workflow quality, and human oversight quality. The NIST AI Risk Management Framework is a strong reference point because it encourages mapping intended use to measurable risk controls rather than treating benchmark scores as sufficient assurance.

In practice, teams often split evaluation into layers:

• Product correctness: does the output satisfy the user task?
• Policy compliance: does it avoid prohibited content, unsafe advice, or privacy violations?
• Operational resilience: does the system behave consistently under prompt injection, malformed input, or adversarial prompts?
• Human override: are there clear stop conditions, review queues, and rollback paths?

Engineers are usually best placed to build the harness, test data pipeline, and reporting automation. PMs and domain experts should decide which failure modes are unacceptable, how much ambiguity is tolerable, and what tradeoffs are justified by business value. Security and assurance teams should validate that evaluation is not overfit to a narrow benchmark set and that red-team findings are incorporated into release criteria. For adversarial AI testing, MITRE’s MITRE ATLAS is helpful for structuring threat-informed scenarios, while the OWASP Top 10 for Large Language Model Applications helps teams turn common failure patterns into test cases.

These controls tend to break down when evaluation ownership sits in a single engineering team because business stakeholders are then asked to approve risk they did not help define.

Common Variations and Edge Cases

Tighter evaluation criteria often increases review time, labeling cost, and release friction, requiring organisations to balance confidence against delivery speed. That tradeoff is unavoidable in higher-risk AI use cases, and there is no universal standard for this yet. A consumer-facing recommendation model may tolerate lower-stakes mistakes than an agent that can send emails, create tickets, or trigger payments.

One common edge case is when domain experts disagree with PMs about acceptable output quality. In those situations, current guidance suggests the tie-breaker should be the risk profile of the use case, not whichever group owns the roadmap. Another edge case is reusable evaluation across multiple models. A shared rubric can improve consistency, but it should not erase context-specific controls such as regional law, customer segment, or data sensitivity. For agentic workflows, the evaluation should also consider tool use, permissions, and escalation logic, because a model that is “accurate enough” in isolation may still be unsafe once it can act.

Where the answer changes most sharply is in regulated or safety-critical environments, especially when the model influences identity decisions, financial outcomes, or customer harm. In those settings, evaluation governance should be documented, auditable, and linked to release authority, not left as an informal engineering convention. That is also where the line between AI quality and security quality disappears, because output correctness, policy compliance, and misuse resistance become the same decision.

Related resources from NHI Mgmt Group

• How should security teams handle AI-generated phishing that looks like normal business mail?
• How do IAM teams decide whether an AI agent should be treated like an NHI?
• How do SOCs decide when to contain an AI agent acting like an insider?
• What does good NHI governance look like for audit and compliance purposes?