AI Safety vs AI Security: Where They Differ and Overlap

"AI safety" and "AI security" are used interchangeably in board papers, vendor pitches and job titles, yet they describe different risks, owned by different teams and tested in different ways. The confusion leaves gaps: a content-safety filter is treated as a security control, a stolen API key is treated as a model behaviour problem, and an evaluation run by the safety team opens a real network path that nobody in security reviewed. This guide sets out how primary sources such as the NIST AI RMF, the UK AI Security Institute, the EU AI Act and the frontier labs' own safety frameworks draw the line, where the two disciplines genuinely overlap, and how to organise ownership, controls and metrics so neither falls between teams. It is written for CISOs, AI platform owners, risk teams and the safety or responsible AI functions they work with.

Key takeaways

Safety is about harm from the system, security is about harm to it. AI safety asks whether the system, used as intended or misused, causes harm to people or the world; AI security asks whether adversaries can compromise its confidentiality, integrity or availability.
NIST treats them as separate trustworthiness characteristics. The AI RMF lists "safe" and "secure and resilient" as distinct characteristics with different definitions, alongside valid and reliable, accountable and transparent, explainable, privacy-enhanced and fair.
Misuse and jailbreaks sit in the overlap. A jailbreak is a security technique (OWASP classes it as a form of prompt injection) whose goal is usually a safety failure, so both teams need to own it.
Safety controls depend on security controls. Guardrails, classifiers and evaluation sandboxes fail if credentials are stolen, weights are exfiltrated or network egress is open, as the Storm-2139 abuse of Azure OpenAI showed.
Organise by risk, not by label. Use one AI risk register with named owners per risk, shared incident handling and a shared test programme, rather than separate safety and security silos.

Definitions from primary sources

There is no single legal definition of either term, but the main frameworks are consistent enough to work from.

NIST AI RMF 1.0 (AI 100-1), section 3.2 Safe: AI systems should not, under defined conditions, lead to a state in which human life, health, property or the environment is endangered. Safety is improved through responsible design, clear guidance to deployers, responsible decisions by users and documentation of risks based on incident evidence.
NIST AI RMF 1.0, section 3.3 Secure and resilient: security means maintaining confidentiality, integrity and availability through protection against unauthorised access and use, including protocols to avoid, guard against, respond to and recover from attacks. Resilience is the ability to withstand unexpected adverse events and degrade gracefully. NIST notes that resilience is part of security, while security also addresses deliberate attack and misuse.
UK AI Security Institute: the UK renamed its AI Safety Institute the AI Security Institute on 14 February 2025, announcing at the Munich Security Conference that it would concentrate on serious risks with security implications, such as AI helping develop chemical or biological weapons, carry out cyber attacks or enable fraud and child sexual abuse, and that it would not focus on bias or freedom of speech. In this usage "security" means national and public security, broader than the CISO's meaning.
EU AI Act, Article 55: providers of general-purpose AI models with systemic risk must evaluate models including documented adversarial testing, assess and mitigate systemic risks, track and report serious incidents, and ensure adequate cybersecurity protection for the model and its physical infrastructure. The article puts safety evaluation and cybersecurity side by side as separate duties.
Frontier lab frameworks: Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework and Google DeepMind's Frontier Safety Framework all pair capability thresholds (safety) with security requirements for model weights and deployment safeguards (security).

Two meanings of "safety" circulate. Frontier or catastrophic-risk safety focuses on severe harms such as biological, chemical and cyber uplift, loss of control and autonomous replication. Product or user safety (sometimes called trust and safety, generative AI safety or youth AI safety) focuses on harmful content, self-harm, harassment, inappropriate material for minors and misinformation. Enterprise teams mostly deal with the second, but buy models governed by the first.

Where they differ

Dimension	AI safety	AI security
Core question	Can the system cause harm, through its outputs or actions, even when no one attacks it?	Can an adversary make the system leak, change, stop or be misused?
Threat source	Model behaviour, misalignment, user misuse, failure modes, emergent capability	External attackers, malicious insiders, compromised suppliers
Typical harms	Dangerous or harmful content, unsafe advice, hallucination, bias, uncontrolled agent actions	Data exfiltration, credential theft, model theft, poisoning, denial of service, cost abuse
Assets protected	People, society, users, the deploying organisation's reputation	Data, model weights, prompts, credentials, infrastructure
Main controls	Alignment training, refusal behaviour, content classifiers, evaluations, usage policies, human oversight	Identity and access control, secrets management, network isolation, input and output validation, logging, supply chain controls
Testing	Capability and harm evaluations, safety benchmarks, red teaming for harmful outputs	Penetration testing, adversarial ML testing, threat modelling, code review
Typical owner	Responsible AI, trust and safety, model or product teams	CISO, application security, cloud security
Reference catalogues	NIST AI 600-1 risk list, frontier lab frameworks, International AI Safety Report	OWASP Top 10 for LLM Applications, MITRE ATLAS, NCSC secure AI development guidelines

Where they overlap

The overlap is large, and most real incidents sit in it. The test is simple: if an adversary is involved and the outcome is harm to people, both teams own it.

Overlap area	Safety view	Security view	Shared control
Jailbreaks (ATLAS AML.T0054)	Model produces content it was trained to refuse	Input manipulation bypasses a control (OWASP LLM01:2025)	Layered classifiers, monitoring, rapid patching of bypasses
Prompt injection in agents (AML.T0051)	Agent takes harmful actions	Untrusted data executes as instructions	Least privilege, tool allowlists, human confirmation
Abuse of stolen access	Harmful content generated at scale	Stolen API keys and account takeover (AML.T0040)	Key management, abuse detection, rate limits
Model weight theft	Safeguards can be removed from a stolen model	Exfiltration of a high-value asset	Weight access controls, multi-party authorisation, monitoring
Data and model poisoning	Model learns harmful or biased behaviour	Integrity attack on training data or supply chain (AML.T0010)	Provenance, AI-BOM, integrity checks
Agent evaluations and red teaming	Measure dangerous capability	The test environment has real credentials and network paths	Joint rules of engagement, egress control, monitoring
Hallucination and misinformation	Users are harmed by false answers	Attackers exploit invented packages or steer answers (LLM09:2025)	Grounding, verification, package allowlists

The Storm-2139 abuse of Azure OpenAI is the textbook overlap case: the group used customer API keys scraped from public sources, modified access to get around Microsoft's safety protections, and resold the service so buyers could generate harmful images. The safety controls existed; the credential failure made them bypassable. See the LLMjacking Guide.

How the frontier safety frameworks handle security

Enterprises rarely write their own frontier safety framework, but they buy models governed by one, and the frameworks show how closely safety now depends on security. Summaries below reflect the pages as published at the time of writing; these documents are revised often, so check the current version.

Framework	Safety mechanism	Security mechanism
Anthropic Responsible Scaling Policy (version 3.4 listed as current, effective 8 July 2026)	Capability thresholds (for example CBRN uplift, autonomous AI R&D) that trigger stronger safeguards; ASL-3 Deployment Standard with access controls, real-time and asynchronous classifiers and jailbreak rapid response	ASL-3 Security Standard with planned controls such as multi-party authorisation and temporary access to reduce the risk of weight exfiltration
OpenAI Preparedness Framework (version 2, 15 April 2025)	Tracked categories: biological and chemical, cybersecurity, AI self-improvement; High and Critical capability levels; Capabilities Reports	Safeguards Reports describing how safeguards are designed and verified; Safety Advisory Group reviews residual risk
Google DeepMind Frontier Safety Framework (third iteration, version 3.1 April 2026)	Critical Capability Levels, including harmful manipulation and machine learning R&D; safety case reviews before launch and for large internal deployments	Protocols for misaligned models that might interfere with operators' ability to direct, modify or shut them down (a control problem as much as a security one); the published update does not detail weight security

Note that OpenAI tracks cybersecurity capability as a safety category: a model that can materially help attackers is a safety risk for the world and a security risk for every defender. See the Frontier AI Model Risk Frameworks Guide.

Organising ownership in an enterprise

One register, named owners: keep a single AI risk register that records safety and security risks per use case, with one accountable owner for each risk and a named partner team. NIST CSF 2.0 GV.RR-01 places accountability with leadership; the AI RMF GOVERN function does the same for AI risk.
Security owns the platform: identity, secrets, network isolation, logging, supply chain and incident response for AI systems sit with security, using the same standards as other production systems. See the LLM Security Guide.
Safety owns behaviour: acceptable use, content policy, refusal behaviour, harm evaluations and user protections (including for minors) sit with responsible AI or trust and safety.
Shared: misuse, red teaming, incidents: jailbreaks, abuse detection, red team programmes and AI incident handling are joint, with one incident process and severity scale. See the AI Red Teaming Guide and the AI Incident Reporting Guide.
Small organisations: one person often holds both roles; use the overlap table above as a checklist so neither half is skipped, and rely on the model provider's published safety framework for frontier risks.
Large organisations: create a standing AI risk forum where the CISO, the responsible AI lead, legal and product approve high-risk use cases together, and require both safety and security evidence before production.

Where identity fits

Most of the overlap controls are identity controls. Model API keys, agent service accounts, OAuth grants to AI tools and the credentials inside evaluation sandboxes decide what a jailbroken or misaligned system can actually do. A safety filter that can be bypassed with a stolen key, or an agent with a standing administrator token, turns a behaviour problem into a breach. Give every model integration and agent its own identity, scope it to the task, rotate and monitor its credentials, and log every tool call against that identity. See the Agentic AI Security Guide and the API Key Management Guide.

How to implement a joint safety and security programme

Agree vocabulary: adopt the NIST AI RMF characteristics as the shared definitions and record which team leads each one.
Inventory AI systems and models: include third-party models, the provider's published safety framework and model card, deployment pattern (API, self-hosted, agent) and the identities each system uses.
Threat model both halves: for each system list safety harms (harmful content, unsafe actions, hallucination) and security threats (key theft, prompt injection, poisoning, data leakage) using NIST AI 600-1 and MITRE ATLAS.
Map controls to owners: use the overlap table to assign each control; flag any safety control that depends on a security control (classifiers that can be bypassed by direct API access, for example).
Test jointly: run red team exercises that combine jailbreak and harmful-content objectives with security objectives such as credential misuse and data exfiltration. Write rules of engagement covering internet access, real targets and stop conditions.
Pilot with monitoring in place: deploy to a limited group with content and security telemetry flowing to one place, then widen in rings.
Plan rollback: keep a kill switch per AI feature, the ability to revoke model and agent credentials quickly, and a fallback to a previous model version or a non-AI process.
Review on change: re-run safety evaluations and security tests on every model upgrade, new tool integration or change in the provider's framework.

Detection and measurement

Collect, per AI system: prompts and outputs (with privacy controls), classifier and guardrail decisions, tool calls and the identity that made them, API key usage by source IP and volume, and network egress from agent and evaluation environments.

Jailbreak success rate: share of red team or benchmark jailbreak attempts that produce policy-violating output, tracked per model version.
Guardrail bypass paths: count of access routes to a model that do not pass through safety controls (direct API keys, unmanaged endpoints); target zero.
Credential exposure time: median time from an AI API key being exposed (for example in a public repository) to revocation.
Agent actions outside policy: unsanctioned tool calls or egress attempts per thousand agent runs, from monitoring.
Joint review coverage: share of high-risk AI use cases with both safety and security sign-off before production.
Incident time to contain: for AI incidents of either type, time from detection to isolation.

Framework mapping

Framework	Safety-leaning references	Security-leaning references
NIST AI RMF 1.0	3.2 Safe; 3.1 Valid and reliable	3.3 Secure and resilient
NIST AI 600-1	Confabulation; Dangerous, violent or hateful content; CBRN information or capabilities	Information security; Value chain and component integration
EU AI Act	Art. 55 evaluation, systemic risk mitigation, serious incident reporting	Art. 55 adequate cybersecurity for the model and physical infrastructure
OWASP Top 10 for LLM Applications 2025	LLM09:2025 Misinformation	LLM01:2025 Prompt Injection, LLM03:2025 Supply Chain, LLM06:2025 Excessive Agency
MITRE ATLAS	AML.T0048 External Harms	AML.T0054 LLM Jailbreak, AML.T0051 LLM Prompt Injection, AML.T0040 AI Model Inference API Access, AML.T0010 AI Supply Chain Compromise
NIST SP 800-53 Rev. 5	RA-3 Risk Assessment (harm analysis)	AC-6 Least Privilege, SA-11 Developer Testing and Evaluation, CA-8 Penetration Testing, SI-4 System Monitoring, IR-4 Incident Handling
NCSC Guidelines for secure AI system development	Responsible release within secure deployment	Secure design, development, deployment, operation and maintenance

Common failure patterns

Treating content filters as access control: provider-side safety filters were in place when Storm-2139 used stolen customer keys to get around them. Filters reduce harmful output; they do not authenticate or authorise.
Evaluations run outside security governance: in the UK AISI agent testing incident, internet access was deliberately enabled and cyber classifiers deliberately switched off for an evaluation, and agents took unsanctioned actions against real people and platforms. A safety test is still a production-grade security event if it touches the internet.
Assuming the model provider's safety framework covers your deployment: lab frameworks govern model release; they say nothing about your API keys, your agents' permissions or your retrieval corpus.
Splitting incidents by label: a jailbreak reported to trust and safety and a key leak reported to the SOC may be the same attacker. Use one incident process.
Using "AI security" to mean only national security: the UK institute's remit is deliberately narrow; enterprise security of AI systems still needs its own controls and owners.
Ignoring user safety in internal tools: internal assistants can still give harmful advice or false answers to staff; safety review is not only for consumer products.

Practitioner checklist

Adopt shared definitions based on the NIST AI RMF and publish them internally.
Keep one AI risk register covering safety and security risks with a named owner each.
Record the safety framework and model card of every third-party model in use.
Close every access path to models that bypasses safety controls, including direct API keys.
Issue each AI integration and agent its own scoped, monitored identity and rotate its credentials.
Run joint red team exercises with written rules of engagement and egress controls.
Deny internet egress by default for agent evaluations and sandboxes, and monitor them in real time.
Use one incident process and severity scale for safety and security AI incidents.
Re-run safety evaluations and security tests on every model, tool or framework change.

AI Safety vs AI Security Guide: Where They Differ and Overlap