How should security teams test AI systems for safety and security separately?

Why This Matters for Security Teams

Safety and security are related, but they are not the same control objective. Safety testing asks whether the AI behaves acceptably for users and the business, including harmful content, bias, and policy adherence. Security testing asks whether an adversary can manipulate the system through prompt injection, malicious instructions, tool abuse, or data exfiltration. Conflating the two creates a false sense of assurance.

This distinction matters because a model can refuse obvious abuse and still be exploitable through indirect prompts, retrieval poisoning, or unsafe tool chains. Security teams also need to understand how test results map to operational risk: a model that passes safety review may still violate trust boundaries once it is connected to internal systems, agents, or external APIs. Guidance from the NIST Cybersecurity Framework 2.0 is useful here because it reinforces that governance, protection, detection, and response all need explicit ownership.

Practitioners often miss the separation between policy compliance and adversarial resilience. In practice, many security teams encounter AI risk only after a benign-looking deployment has already been connected to sensitive data or privileged tools, rather than through intentional pre-release attack testing.

How It Works in Practice

The most reliable approach is to maintain two distinct test plans, two evidence sets, and two sign-off paths. Safety evaluation should measure whether the model stays within content and behavior boundaries under normal and edge-case prompts. Security evaluation should simulate hostile inputs, compromised context, and tool-chain abuse. These tests overlap at the edges, but the evidence they produce serves different decisions.

For safety, teams usually test refusal behavior, harmful advice generation, protected-class bias, and consistency with product policy. For security, the focus shifts to adversarial prompts, jailbreak attempts, hidden instructions in retrieved content, exfiltration attempts, and whether the model can be pushed into unauthorised actions. The same scenario can be used in both tracks, but the scoring criteria should differ. Safety asks, "Did the answer violate policy or cause harm?" Security asks, "Did the system preserve boundaries under attack?"

Good practice is to evaluate the full execution path, not just the model output. That means reviewing system prompts, retrieval sources, connectors, function calls, memory, and downstream actions. Where an AI agent can call tools, security testing should verify that approvals, rate limits, scope controls, and logging still hold when the model is influenced by a malicious prompt. MITRE ATLAS is useful for organizing adversarial techniques, while the OWASP Top 10 for Large Language Model Applications helps teams structure common failure modes such as prompt injection and insecure output handling.

• Build a safety test set with policy violations, harmful completions, and fairness edge cases.
• Build a security test set with prompt injection, data leakage, tool misuse, and indirect attacks.
• Test against the model, the retrieval layer, and the agent/tool layer separately.
• Record evidence for both pass/fail outcomes and partial degradations.

For governance, this usually belongs in the release gate alongside change management, logging, and incident response readiness. The NIST AI Risk Management Framework is a strong reference for structuring these evaluations across mapping, measuring, and managing risk. These controls tend to break down when models are deployed with live tools and broad data access before red-team findings have been remediated, because the test environment no longer matches production privilege.

Common Variations and Edge Cases

Tighter evaluation often increases delivery time and review overhead, requiring organisations to balance launch speed against the cost of a risky rollback. That tradeoff is especially visible when teams want one scorecard for both safety and security, but current guidance suggests that combined scoring often hides critical differences in attack surface.

There is no universal standard for how much test coverage is enough, so teams should define thresholds by use case. A customer-facing chatbot may need stronger safety thresholds, while an internal agent with API access may need deeper security testing even if its content moderation score is excellent. Where regulated data is involved, security testing should also check for accidental disclosure through logs, retrieval, or tool outputs.

Edge cases matter most when the AI sits inside a workflow rather than as a standalone chat interface. RAG systems can pass safety checks but still leak sensitive context if retrieval is poisoned. Agentic systems can behave politely while still executing unsafe actions through a legitimate tool. In those environments, best practice is evolving toward layered testing of model behavior, orchestration logic, and identity or privilege boundaries together. For security teams, that means treating safety as a quality gate and security as a trust boundary test, not interchangeable labels.

Related resources from NHI Mgmt Group

• How should security teams test for visual prompt injection in multimodal AI systems?
• How should security teams evaluate AI systems that refuse to cooperate with safety testing?
• How should security teams test AI systems beyond output filtering?
• How should security teams test multimodal AI systems before production?