The failure is that benchmark performance and safety readiness are being treated as the same thing. A model can answer well on standard tasks and still be steerable into unsafe behaviour through role-play, prompt injection, or context shifts. Organisations need separate approval criteria for capability, refusal behaviour, and runtime containment.
Why Benchmark Success Can Hide a Safety Failure
Benchmarks usually test whether a model can produce the right answer under controlled prompts. Jailbreakability tests something different, whether the model can be induced to ignore refusal boundaries when the conversation changes shape. The failure is treating capability as proof of safe deployment, when the real issue is whether the model resists manipulation in realistic interaction patterns.
That gap matters because benchmark scores often optimise for correctness, consistency, or task completion, while safety failures appear under adversarial framing, role-play, prompt injection, or prompt drift. A model can look excellent in evaluation and still be fragile at runtime once the user changes persona, adds context, or embeds instructions inside content it is asked to process. For that reason, capability and safety need separate acceptance criteria.
What Actually Breaks Under Jailbreak Pressure
The thing that fails is not only the model output, but the assumption that a single score can represent both utility and control. In practice, the model may still have strong latent capability, but its refusal policy, instruction hierarchy, or context handling is weak enough that an attacker or curious user can steer it into disallowed behaviour. That is why red-team results often diverge from benchmark results.
In security terms, this is a boundary-control failure. The model may follow the most recent, most persuasive, or most contextually salient instruction rather than the intended policy. Red Teaming AI Agents for Identity Abuse is relevant here because the same pattern of role-play, delegation abuse, and approval bypass shows how an apparently capable system can still be manipulated into unsafe action.
That is also why organisations should not equate benchmark approval with production readiness. NIST AI Risk Management Framework supports the idea that trustworthy AI requires separate governance of capability, safety, and ongoing monitoring, not a one-time pass/fail on task accuracy.
How to Evaluate Safety Separately from Capability
The practical answer is to evaluate three things independently: what the model can do, what it refuses to do, and what it can be made to do under stress. Those are different properties. A model that performs well on standard benchmarks should still be tested with jailbreak suites, prompt-injection scenarios, and conversation-chaining that simulates how real users or attackers manipulate context.
Agentic AI Identity Maturity Model is useful as a planning aid because it frames AI systems as governed entities with maturity differences, not as interchangeable chat interfaces. That perspective helps teams decide when approval should depend on containment, observability, and policy enforcement rather than raw model performance.
For the runtime side, NIST Cybersecurity Framework 2.0 is a sensible companion when you need to map model deployment controls, monitoring, and response ownership. The key operational point is to treat safe behaviour as a continuously managed control objective, not an intrinsic property that is proven once by a benchmark report.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI Risk Management Framework | Benchmark success versus jailbreak resistance is an AI trust and risk-governance issue. |
| GOVERN — Govern | Governance must distinguish performance metrics from safety readiness. | |
| Recommendation — Assess capability, refusal robustness, and monitoring as separate acceptance criteria. Set policy that benchmark performance does not equal deployment approval. | ||
| NIST CSF 2.0 | GV.OV-01 — Oversight of the cybersecurity risk management strategy | Safety approval needs oversight beyond a single benchmark score. |
| PR.AA-01 — Identities and credentials are managed for authorized access | Runtime containment depends on controlling who and what can act in the system. | |
| Recommendation — Define separate approval gates for capability, safety, and runtime containment. Restrict model, tool, and operator access to the minimum required. | ||
Practitioner Guidance
What to verify: Require separate evidence for task performance, refusal robustness, and runtime containment before a model is approved for broader access. If a model is strong on benchmark tasks but weak under adversarial prompting, treat it as capable but not yet safe enough for unconstrained use.
Decision rule: If the model will be exposed to user-supplied content, multi-turn interaction, or tools, assume benchmark scores understate the real risk and add adversarial evaluation before deployment. If the model is isolated, low-impact, and heavily mediated, the acceptance bar can be narrower, but only after the containment assumption is explicit.
Common mistake: Teams often promote a model because it “passed evaluation” when the evaluation only measured answer quality. The better question is whether the model stays within policy when instructions conflict, context is polluted, or a user tries to social-engineer the conversation.
Practitioner takeaway: A model that jailbreaks easily has failed the safety boundary, even if it scored well on benchmarks, so approval should hinge on adversarial robustness and containment rather than headline capability alone.