AI safety and AI security are different problems with different controls

At a glance

What this is: This is an analysis of why AI safety and AI security must be treated as separate governance problems, with examples showing how model behaviour, agent access, and adversarial manipulation fail in different ways.

Why it matters: It matters because IAM, PAM, and security teams now have to govern AI systems as runtime actors with access to tools, data, and credentials, not just as text generators.

By the numbers:

• IBM's 2025 report puts the global average cost of a data breach at $4.44M.

👉 Read Promptfoo's analysis of AI safety versus AI security in LLM systems

Context

AI safety and AI security are not interchangeable control problems. Safety is about preventing harmful model outputs during normal operation, while security is about defending the LLM stack, connected tools, and data from adversarial misuse. The article's primary keyword, AI safety and AI security, sits at the centre of the governance gap many teams still leave unresolved.

That distinction matters because once an LLM is connected to code execution, databases, APIs, or MCP-connected tools, it becomes part of the access-control plane as well as the content plane. For identity teams, the issue is no longer only what the model says. It is who or what is allowed to act through the model, under what privileges, and with what auditability.

The examples in the article are consistent with a broader industry pattern: organisations often move faster on model enablement than on runtime containment. That is atypical only in scale, not in structure, because the same governance mistake now repeats across agents, plugins, and connected enterprise workflows.

Key questions

Q: How should security teams test AI systems for safety and security separately?

A: Run two evaluation tracks. Safety tests should measure harmful content, bias, refusal quality, and policy compliance. Security tests should focus on prompt injection, hidden instructions, tool misuse, data leakage, and unauthorised actions. If a system is only tested for one dimension, teams can ship a model that sounds safe while still being easy to manipulate.

Q: Why do AI agents complicate existing IAM and PAM controls?

A: AI agents complicate IAM and PAM because they often inherit delegated credentials, operate across multiple systems, and keep acting after the initial approval moment has passed. Human session assumptions, periodic reviews, and static privilege models do not reflect that behaviour. The result is a governance gap between what was granted and what the agent can actually do.

Q: What breaks when prompt injection is not governed like an access problem?

A: The organisation may treat malicious text as a harmless message, even though it can steer an agent into exposing data or taking privileged actions. Prompt injection is dangerous because it turns untrusted content into a control plane for behaviour. Teams need policy and authorisation checks around outputs, not just message filtering.

Q: Which controls matter most when an LLM can access production systems?

A: Prioritise least privilege, short-lived credentials, tool-level allowlists, immutable logging, and human approval for irreversible actions. Those controls matter more than output filtering alone because the main risk is not just what the model says. It is what the model can be induced to do through authorised integrations.

Technical breakdown

Why AI safety and AI security fail in different ways

AI safety controls are designed to shape the model's outputs, usually through alignment training, refusal behaviour, and content filtering. AI security controls protect the system boundary around the model, including prompts, tools, credentials, memory, and connected data sources. A model can be safe enough to avoid harmful speech and still be insecure if an attacker can coerce it into leaking secrets, running code, or invoking APIs. The article's point is that identical interaction surfaces can produce different failures depending on whether the threat is harmful content or adversarial intent.

Practical implication: test both the model's output behaviour and the access paths around it, because one without the other leaves a real control gap.

How prompt injection turns trusted content into a control path

Prompt injection works because LLMs often cannot reliably distinguish instructions from data. External content such as emails, calendar items, README files, or web pages can contain hidden commands that the model treats as legitimate input. When the model is connected to tools or memory, those injected instructions can trigger actions outside the user's intent. This is an identity and authorisation problem as much as a model problem, because the model may inherit the privileges of the session or integration without enough contextual validation.

Practical implication: isolate external content, constrain tool permissions, and treat retrieved data as untrusted until it is validated.

Why autonomous agents create a credential and privilege problem

AI agents are not just models that talk. They are systems that can choose actions, call tools, and sequence work across sessions. That creates a non-human identity issue, because the agent may hold credentials, API keys, or delegated access while operating with limited human oversight. If the agent can write to production systems, manipulate logs, or call downstream services, then its privilege scope becomes a security boundary. The article shows that the most serious failures emerge when autonomy is paired with standing access and insufficient containment.

Practical implication: govern agents like privileged non-human identities, with scoped access, short-lived credentials, and strong audit trails.

Threat narrative

Attacker objective: The attacker wants the AI system to reveal secrets, execute unauthorised actions, or disrupt production workflows while appearing to behave normally.

1. Entry occurs through trusted content or connected integrations, such as a prompt, calendar item, README file, or other external data source that the model reads during normal operation.
2. Escalation follows when the model accepts the hidden instruction as a valid command and uses its connected tools, credentials, or session privileges to act on it.
3. Impact occurs when the model leaks data, executes unauthorised actions, or changes system state in ways the user never intended.

NHI Mgmt Group analysis

AI safety and AI security are separate governance disciplines, not two labels for the same risk. Safety failures create harmful outputs, but security failures create unauthorised action paths through tools, data, and credentials. Organisations that collapse the distinction tend to overinvest in content filters while underinvesting in access control and runtime containment. The right governance model is to treat model behaviour and system privilege as different control surfaces.

AI agents are becoming non-human identities whether teams name them that way or not. Once an agent can authenticate, call APIs, or touch data stores, it behaves like a workload identity with agency. That means IAM and PAM teams have to care about delegated access, scope, session lifetime, and auditability, not just model quality. The control question is no longer whether the model is helpful. It is whether its authority is bounded enough to contain misuse.

Prompt injection creates a trust boundary failure between data and instruction. The article shows why external content cannot be assumed safe simply because it arrives through an approved workflow. That is a named concept worth carrying forward: instruction-data confusion. Practitioners should recognise it as a governance gap in retrieval pipelines, plugin chains, and agent tool use, not as a quirky model bug.

Security testing for AI must be continuous because the attack surface changes with every integration. A static approval process cannot keep pace with new tools, new prompts, new memory stores, and new agent behaviours. That is why model red teaming, runtime controls, and identity governance need to operate together. The broader lesson is that AI enablement without operational guardrails becomes an access-risk programme in disguise.

What this signals

Instruction-data confusion is becoming a practical governance problem for any programme that feeds external content into an LLM. The reader takeaway is simple: every retrieval pipeline, plugin chain, and agent workflow should be treated as a potential command path until validated. For standards alignment, map those controls to the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework.

Identity teams should expect AI agents to drive new governance demand around delegated access, credential scope, and revocation speed. The practical pressure point is not model accuracy but access authority, which means PAM, IAM, and security architecture teams need a shared inventory of which agents can touch which systems. This is where agent identity becomes a control design issue, not a taxonomy debate.

The operational signal to watch is whether your AI programme can prove what each agent read, what it called, and what it changed. If you cannot reconstruct that sequence, you do not have effective auditability. That makes incident response slower and compliance evidence weaker, especially when external content is the injection vector.

For practitioners

• Separate safety testing from security testing Build distinct evaluation tracks for harmful outputs and adversarial manipulation. Measure refusals, bias, and toxicity separately from prompt injection, data leakage, and unauthorised tool use.
• Treat AI agents as privileged non-human identities Assign scoped, short-lived credentials to every agent that can call tools or access data. Map each agent to an owner, an intended function, and a revocation path.
• Sandbox external content before the model can act on it Strip executable instructions from emails, calendar items, README files, and retrieved pages before they reach the model context. Keep untrusted content separate from control tokens and tool calls.
• Log every tool call and data access event Create immutable audit records for prompts, tool invocations, credential use, and data reads or writes. That evidence is essential for containment, forensics, and compliance review.

Key takeaways

• AI safety and AI security fail differently, so teams need different tests and different controls.
• Autonomous agents create non-human identity and privilege problems that conventional model guardrails do not solve.
• Prompt injection turns trusted content into an execution path unless data, tools, and credentials are tightly separated.

Key terms

• AI Safety: AI safety is the discipline of preventing an AI system from taking unintended or harmful actions on its own. It focuses on the behaviour the system generates, even when no external attacker is involved. For identity teams, safety is about limiting what the agent can do once it is already operating.
• AI security by design: AI security by design means building security, privacy, and access controls into AI systems from the start instead of adding them after deployment. In practice, it combines data governance, human oversight, documentation, and continuous monitoring so that model behaviour is auditable and bounded.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Agent Identity: An agent identity is the set of attributes, credentials and permissions assigned to an autonomous software entity. It is treated as a non-human identity because it can authenticate, act on systems and accumulate access over time, which creates governance, audit and lifecycle obligations similar to other production identities.

What's in the full article

Promptfoo's full analysis covers the operational detail this post intentionally leaves for the source:

• Concrete incident references and evaluation examples for Replit, Grok, Gemini, Cursor, and Amazon Q
• Practical test design for separating safety red teaming from security red teaming
• Implementation guidance for prompt sanitisation, tool containment, and immutable logging
• A detailed breakdown of OWASP LLM Top 10 mappings and related control choices

👉 Promptfoo's full article covers the incident examples, control patterns, and testing approach in more depth.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management. It gives security practitioners a shared foundation for governing machine access, delegated authority, and auditability across modern identity programmes.