AI safety and AI security are not the same governance problem

At a glance

What this is: This analysis separates AI safety from AI security and shows that the two problems have different failure modes, owners, and readiness criteria.

Why it matters: IAM and security teams need this distinction because AI systems now sit inside identity, access, and tool-use workflows, where governance failure can mean either harmful but intended behaviour or adversarial abuse.

Context

AI safety and AI security are not the same governance problem. Safety deals with harmful outcomes from a system acting as designed, while security deals with adversaries steering the system off course through prompt injection, poisoned data, compromised assistants, or hostile tool integrations.

That split matters because AI systems increasingly operate inside identity-rich workflows, including developer environments, service access paths, and delegated tool use. If governance does not distinguish intended behaviour from attacker-induced behaviour, ownership, incident handling, and readiness criteria all become blurred.

Key questions

Q: How should security teams separate AI safety failures from AI security failures?

A: Treat safety as a question of whether the system’s intended behaviour is acceptable, and security as a question of whether an attacker can redirect that behaviour. Give each risk class its own owner, control set, and production gate. If one review process covers both, accountability becomes vague and response decisions slow down.

Q: Why do AI coding assistants increase AppSec governance risk?

A: They increase risk because they optimise for speed and plausible output, not for context-aware security validation. In practice, that means developers can accept more code with less scrutiny, which raises the chance of vulnerable functions, exposed secrets, and compliance failures entering the repository and CI/CD pipeline.

Q: What breaks when AI security is treated only as model security?

A: Model-only security misses the part of the system that actually touches tools, data, and workflows in production. A secure model can still produce unsafe outcomes if the surrounding agent, connectors, or permissions are not governed. Practitioners need controls that follow the operational identity, not just the model artefact.

Q: What should organisations do when AI agents and build tools have access to code, secrets, and deployments?

A: They should treat those agents as governed non-human identities. Inventory their permissions, separate creation from approval, and limit their access to the smallest set of repositories, registries, and deployment paths needed for the task.

Technical breakdown

Why safety and security fail for different reasons

AI safety asks whether a system can cause harm even when nothing is attacking it. That includes bias, manipulation, unsafe recommendations, and undesirable downstream effects from a model behaving exactly as permitted. AI security asks whether an adversary can change what the system does by altering inputs, data, tooling, or environment. The distinction matters because the first is about allowed behaviour and acceptable harm, while the second is about hostile control of the AI stack.

Practical implication: Separate safety reviews from security threat modelling so each problem has its own control criteria and sign-off path.

How prompt injection, data poisoning, and malicious MCP servers change the attack surface

Modern AI systems expose a wider attack surface than a static model. Prompts can be manipulated, training or retrieval data can be poisoned, coding assistants can be compromised, and MCP servers can become a control channel into tools and data sources. These are security problems because the attacker is not asking the model to be unsafe in the abstract. They are using infrastructure, identity, and tool delegation to redirect behaviour for their own ends.

Practical implication: Treat AI toolchains as part of the attack surface and apply identity, logging, and trust controls to every connected component.

Why identity and delegated access are now part of AI governance

Once an AI system can call tools, consult services, or act inside a workflow, its security posture depends on what it can reach and under what identity. That makes access scope, credential handling, and tool authorization core design issues rather than afterthoughts. In practice, an AI agent or assistant can become a non-human actor with delegated privileges, which means classic governance questions about who can do what now apply to the system itself.

Practical implication: Define and review AI tool permissions the same way you would review privileged access for any other delegated actor.

Threat narrative

Attacker objective: The attacker aims to bend an AI system into leaking data, executing unwanted actions, or amplifying harmful outcomes while appearing to operate normally.

1. Entry occurs when an attacker uses prompt injection, poisoned data, or a compromised extension to reach the AI workflow.
2. Credential or tool access is abused when the assistant or agent is allowed to invoke connected systems with excessive trust or scope.
3. Impact follows when the AI is steered into exfiltrating secrets, running destructive commands, or producing harmful outputs at scale.

Breaches seen in the wild

• MemTensor MemoryOS supply chain attack 2026: Stolen CI publish tokens let attackers ship a credential-stealing worm in MemTensor's AI agent memory packages on npm and PyPI.
• 12,000 secrets in LLM training data: Truffle Security found 11,908 live API keys and passwords hard-coded in web pages captured by Common Crawl, a dataset used to train LLMs.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI safety and AI security must be governed as distinct risk classes: conflating them creates false confidence and broken ownership. Safety asks whether an approved system can still cause unacceptable harm, while security asks whether an attacker can redirect that system through hostile inputs, poisoned data, or compromised tools. Organisations that merge the two end up with vague accountability and inconsistent readiness criteria. The practical conclusion is that safety and security need separate control objectives, even when they share the same platform.

Agentic AI turns delegated access into a governance problem, not just a technical one: once a model can invoke tools, consult services, or act inside a workflow, its permissions become part of the security boundary. That is an identity question as much as an AI question, because the system is now acting through a non-human identity or delegated trust chain. Enterprises that ignore that shift will struggle to explain who authorised what, under which controls, and with what rollback path. The practitioner implication is to treat AI delegation as privileged access.

Model risk language is no longer enough for operational AI governance: the article’s core contribution is to separate harmful behaviour from adversarial control, which is the governance split most enterprises still miss. A model can be secure and unsafe, or safe in principle and insecure in practice, and those are different remediation paths. That makes AI governance debt a real category of exposure when roles, approvals, and incident playbooks are written as if one control layer covers both. Security teams should insist on explicit ownership for each risk path.

Development environments are now identity and security boundary zones: the article correctly points to IDEs, coding assistants, and MCP ecosystems as places where AI risk becomes operational. Those environments combine credentials, code, and delegated execution, which means a compromise can cross from productivity tooling into production impact. The important takeaway for practitioners is that AI security cannot be treated as a policy overlay. It needs the same access control, logging, and containment discipline applied to any high-trust execution path.

Privacy sits adjacent to both safety and security, but it is not either one: data collection, inference, retention, and disclosure create confidentiality exposure, while safety concerns persist even when privacy and security controls are technically intact. That means enterprises cannot use a privacy programme as a substitute for AI governance, or vice versa. The control model has to answer three separate questions: what data may be used, what an attacker could force the system to do, and whether the system’s intended behaviour is acceptable. Practitioners should design all three explicitly.

From our research library:

• Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.

What this signals

AI governance debt is now a practical risk: when safety, security, and privacy are discussed as one problem, organisations usually end up with blurred ownership and mismatched controls. The result is not just weak policy language. It is a programme that cannot tell whether it is responding to harmful behaviour, hostile manipulation, or data exposure until after the incident.

Non-human actors need identity boundaries: once assistants and agents can call tools or reach services, they stop being a generic feature and start behaving like delegated actors inside the enterprise. That means permission scope, logging, and revocation logic need to move closer to issuance time, not review time. For identity teams, this is the point where AI governance becomes an access governance problem as well.

Development environments are becoming the new control point: if coding assistants, MCP servers, and extensions can reach repositories or production systems, then the IDE is part of the attack surface. The programme response is to bring those workflows under the same containment, monitoring, and approval discipline used for other high-trust execution paths.

For practitioners

• Separate safety sign-off from security sign-off Create distinct approval criteria for harmful intended behaviour and adversarial compromise, then assign each to a named owner before production use.
• Inventory AI-connected tools and credentials Map every assistant, agent, MCP server, and extension to the data and systems it can reach, including secrets and service accounts.
• Constrain delegated permissions for AI systems Limit AI tool access to the minimum scope needed for the task, and review whether the identity used by the system is tightly bounded.
• Test AI workflows for hostile input paths Red-team prompts, retrieval sources, and tool calls to see whether injected instructions can trigger secret exposure or unsafe actions.
• Add AI incidents to existing response playbooks Define how to contain compromised assistants, revoke delegated access, and preserve logs when an AI toolchain is manipulated.

Key takeaways

• AI safety and AI security fail in different ways, so they need different ownership, controls, and sign-off criteria.
• The biggest governance shift is that AI systems now act through delegated access and connected tools, which makes identity and authorization central.
• Enterprises should treat prompts, toolchains, and assistants as part of the attack surface, not as isolated productivity features.

Key terms

• AI Safety: AI safety is the discipline of preventing an AI system from taking unintended or harmful actions on its own. It focuses on the behaviour the system generates, even when no external attacker is involved. For identity teams, safety is about limiting what the agent can do once it is already operating.
• AI Security Platform: An AI security platform governs how people and agents use AI systems across prompts, responses, files, and tool calls. It goes beyond traditional content filtering by adding intent-aware policy, runtime enforcement, and audit linkage so the organisation can control both the conversation and the action that follows.
• AI Governance: AI governance is the set of controls used to discover, classify, approve, restrict, monitor, and revoke AI-enabled access. It connects identity, data, and policy so organisations can manage what AI can reach, what it can share, and when it should be stopped.
• MCP Server: An MCP server is a tool endpoint that connects an AI agent to external systems and data sources through Model Context Protocol. Because it extends what the agent can reach, it becomes part of the identity and access surface and must be reviewed like any other privileged connector.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It helps security and identity teams build the control language needed for delegated access and AI governance.