Teams should block a release when evaluation results are tied to user-facing quality, compliance exposure, or costly downstream decisions. If a bad output can affect customers, operations, or sensitive data handling, evaluation should operate as a release control, not a reporting layer. The key test is whether the team would accept the same failure in production twice.
Why This Matters for Security Teams
Release gating is not just a model quality question. It is a control decision about whether the organisation is willing to ship a known level of AI risk into production. For LLMs, the impact is often indirect at first: a harmless-looking hallucination becomes a billing error, a bad recommendation becomes an unsafe operational step, or an untrusted tool call becomes a security incident. Current guidance from the NIST AI Risk Management Framework treats this as an enterprise risk issue, not a model benchmark exercise.
Teams often get this wrong by treating evaluation as a dashboard for model developers while product and security leaders assume someone else will make the release call. That separation breaks down when the model is embedded in workflows with compliance, customer, or privilege implications. If the failure mode can create audit exposure, trigger a support cascade, or influence sensitive decisions, evaluation should define a launch threshold before the system reaches users.
In practice, many security teams encounter the need for blocking criteria only after a model has already influenced production decisions, rather than through intentional release governance.
How It Works in Practice
Blocking criteria should be derived from the business and security consequences of a failure, then mapped to evaluation categories that can be measured consistently. That usually means combining quality checks, safety checks, and security checks into one release decision, rather than treating them as separate approval lanes. For agentic or tool-using systems, the bar should be higher because a single bad output can become an external action. The OWASP Agentic AI Top 10 and the NIST AI 600-1 Generative AI Profile both reinforce the need to test for misuse, unsafe outputs, and deployment context before release.
- Set explicit block conditions for high-impact failure modes such as policy violation, data leakage, unsafe instructions, or unreliable tool invocation.
- Define severity thresholds in advance, including which issues require immediate stop, which allow limited rollout, and which only trigger remediation.
- Separate “model quality” from “system risk” by evaluating prompts, retrieval data, tools, memory, and post-processing together.
- Require sign-off from the owner of the business process affected by the model, not only the engineering team.
- Keep a rollback path ready so a blocked release can be reversed quickly without improvising under pressure.
For adversarial testing, teams should also use threat-oriented evaluation such as the MITRE ATLAS adversarial AI threat matrix and, where agent behaviour is central, the CSA MAESTRO agentic AI threat modeling framework. These help teams distinguish ordinary bad answers from exploitable failure patterns that should block release. These controls tend to break down when evaluation is performed only on static prompts because real-world failures emerge from tool access, retrieval drift, and chained actions.
Common Variations and Edge Cases
Tighter release gating often increases delivery overhead, requiring organisations to balance safety against shipping speed and model iteration frequency. That tradeoff is real, especially for teams running frequent prompt or retrieval changes. Current guidance suggests not every defect should block every release; best practice is evolving toward risk-tiered thresholds rather than a single universal pass-fail rule.
Low-risk internal assistants may tolerate more variance than customer-facing or regulated workflows, but that does not mean the former can skip evaluation. The key distinction is whether failure creates reversible inconvenience or irreversible harm. A model that drafts internal notes may merit warning-level issues, while a model that approves refunds, changes access, or supports compliance decisions should block on narrower evidence of instability. If the system can reach sensitive data, the question becomes part of broader identity and access governance as well, especially where human approval is assumed but not technically enforced.
Where there is no universal standard for this yet, teams should document the rationale for every blocking threshold, then revisit it after incidents, audits, or major model updates. That makes the release rule defensible even when the model stack changes faster than the policy cycle.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, MITRE ATLAS and CSA MAESTRO address the attack and risk surface, while NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI risk decisions should reflect enterprise impact, not just benchmark scores. | |
| NIST AI 600-1 | GenAI profile guides testing for unsafe outputs, misuse, and deployment context. | |
| OWASP Agentic AI Top 10 | Agentic systems need blocking rules for tool misuse, unsafe actions, and prompt attacks. | |
| MITRE ATLAS | Adversarial AI testing helps separate defects from exploitable attack patterns. | |
| CSA MAESTRO | MAESTRO supports threat modeling for tool-using and autonomous AI workflows. |
Use ATLAS scenarios to decide which failures indicate exploitable risk and must stop release.
