What do security teams get wrong about model safety testing?

Why This Matters for Security Teams

Model safety testing is often treated as a checkpoint, but it is really a control validation problem. A model can look safe in a lab while still failing under prompt injection, retrieval abuse, tool misuse, or adversarial inputs. Security teams also tend to separate model testing from runtime governance, even though deployed behaviour depends on data sources, identity scopes, and downstream actions. That is why safety results should be interpreted as evidence for a specific context, not as a blanket assurance.

This matters because failures are not limited to harmful text generation. Unsafe outputs can trigger fraud, data leakage, privilege misuse, or broken decision support once the model is embedded in workflows. The NIST Cybersecurity Framework 2.0 is useful here because it reinforces that governance, identification, protection, detection, response, and recovery must all be aligned to operational risk, not just model quality. In practice, many security teams encounter model-safety gaps only after an agent has already been granted access to a tool chain, rather than through intentional pre-production testing.

How It Works in Practice

Effective model safety testing should be layered, scenario-based, and tied to deployment context. The first layer checks the model itself for known weakness classes: toxic completion, hallucination under uncertainty, refusal failures, data leakage, and susceptibility to prompt injection. The second layer evaluates the surrounding system: retrieval sources, system prompts, tool permissions, identity bindings, logging, human approval steps, and fallback behaviour. The third layer examines whether the model remains safe when adversaries vary the input, the conversation state, or the sequence of tool calls.

For practitioners, that usually means combining red-teaming, regression tests, and policy checks with production-like integration testing. The MITRE ATLAS knowledge base is helpful for mapping attack techniques that target machine learning systems, while OWASP guidance for LLM applications is useful for covering issues such as prompt injection and insecure output handling. Where agents are involved, NIST AI Risk Management Framework thinking should be extended to tool authorization and action containment, because model safety is no longer just about what the model says, but what the system allows it to do.

• Test the base model separately from the full application stack.
• Validate safety across benign, ambiguous, and adversarial prompts.
• Check tool access, identity scope, and approval gates before release.
• Measure whether safety controls still work after prompt or retrieval changes.

Current guidance suggests safety testing should be treated as continuous assurance, not a one-time pre-launch event. These controls tend to break down when teams rely on static benchmark suites in rapidly changing RAG and agentic environments because the model’s real risk surface is defined by live data, live permissions, and live actions.

Common Variations and Edge Cases

Tighter safety testing often increases release friction and evaluation cost, requiring organisations to balance stronger assurance against delivery speed. That tradeoff is especially visible when teams need to test both model behaviour and the surrounding orchestration layer. There is no universal standard for this yet, so many programmes adopt a risk-based approach: higher scrutiny for systems that can access sensitive data, invoke tools, or influence customer decisions.

One common edge case is the “safe model, unsafe system” problem. A model may pass content filters, yet still produce harmful outcomes through a weak retrieval source, overbroad connector, or poorly governed agent workflow. Another is domain drift: tests written for a generic chatbot often miss failures in regulated, operational, or multilingual environments. Where the model is used for identity workflows, security teams should also check whether the system can expose credentials, overstep NHI boundaries, or impersonate authorised activity through tool chaining. Emerging practice is to include these identity and action risks in safety review, but current guidance suggests that maturity varies widely across organisations.

For governance-heavy environments, the main mistake is not technical optimism but scope error. Teams certify the model, while the actual risk sits in the integrated service, the permissions model, or the human review process. That is why safety evidence should be versioned, reproducible, and tied to the exact deployment state being assessed.

Related resources from NHI Mgmt Group

• What do security teams get wrong about AI safety testing?
• What do security teams get wrong about model safety filters?
• What do security teams get wrong about vendor access in public safety environments?
• What do security teams get wrong about AI-assisted webpage safety checks?