AI safety and AI security are different problems with different controls

At a glance

What this is: This article separates AI safety from AI security and shows that conflating them leaves LLM deployments with untested harmful-output risks and exploitable attack paths.

Why it matters: IAM, NHI, and AI security teams need the split because helping the model behave safely does not prevent prompt injection, unauthorized tool use, or exposed credentials.

Context

AI safety and AI security solve different control problems. Safety is about preventing harmful model behavior during normal operation, while security is about preventing adversaries from turning the model, its tools, or its data into an attack surface. For teams shipping LLM features, the distinction matters because the same architecture can be safe enough in one dimension and dangerously exposed in the other, especially when access, prompts, and external data are mixed together.

The article’s core governance issue is not model quality alone, but boundary failure. Once an LLM can call tools, read external content, or act on behalf of a user, the question becomes who controls the action path, who reviews the outputs, and what is logged when the system behaves unexpectedly. That is where AI governance intersects with identity, privilege, and machine access control.

Key questions

Q: How should security teams test LLM applications for runtime security failures?

A: Test the full runtime path, not just the model in isolation. Use adversarial prompts, indirect injection through retrieved content, and tool-call scenarios to see whether the application can be steered into unsafe behaviour once retrieval, agents, and external APIs are active. That is where most LLM risk actually appears.

Q: Why do APIs become a bigger security issue when AI agents consume them?

A: APIs become a bigger security issue because agents do not compensate for ambiguity, broken contracts, or overbroad scopes the way humans often do. They repeat the same action at machine speed, so a small trust error can become a large blast radius. Governance has to focus on identity, action scope, and change control, not just connectivity.

Q: What are the signs that prompt injection is bypassing AI governance?

A: Warning signs include unexpected tool calls, answers that ignore prior policy, sudden disclosure of sensitive fields, and citations that look valid but do not support the output. Those symptoms show the model has accepted hostile instructions or untrusted context as if it were trusted input.

Q: Should organisations treat AI as an application or as an identity?

A: Treat it as an identity when the AI can access data, invoke tools, or participate in workflows that affect business systems. That framing makes least privilege, just-in-time access, and lifecycle governance relevant. If you keep treating it only as an application, you will miss the access and delegation behaviours that actually create risk.

Technical breakdown

Safety controls vs security controls in LLM deployments

AI safety controls shape what the model is allowed to say or recommend. They include alignment tuning, output filtering, and policies that reduce bias, toxicity, or dangerous advice. AI security controls protect the surrounding system, which includes prompts, connectors, tool permissions, API credentials, and external data sources. A system can pass safety checks while still being vulnerable to prompt injection or data exfiltration because the model is acting exactly as trained, but within an unsafe trust boundary. That is why testing must cover both model behavior and the runtime environment around it.

Practical implication: Separate content-safety evaluation from adversarial security testing so one control set does not mask failure in the other.

Why tool access changes the risk model for AI agents

Once an LLM or agent can execute code, query databases, or call APIs, it stops being a text-only assistant and becomes a privileged workflow participant. That creates a machine identity and authorization problem, not just a model-quality problem. If external content can influence instructions, the system may treat untrusted data as trusted intent. In practice, the failure often sits in weak privilege separation, broad tool scope, and missing input validation around prompts, files, or messages that enter the agent’s context.

Practical implication: Limit tool scope, isolate untrusted inputs, and govern agent credentials as operational access rather than loose application settings.

Indirect prompt injection and hidden instruction risk

Indirect prompt injection happens when malicious instructions are embedded in content the model is expected to read, such as email, calendar invites, README files, or support tickets. The model sees data and instruction in the same context window, so helpfulness can override the boundary between them. That is why hidden instructions can trigger data disclosure, unwanted actions, or unauthorized commands even when the original user prompt is harmless. This is a security problem because an attacker is manipulating system behavior through untrusted external content.

Practical implication: Treat every external input as hostile until sanitized, labeled, and constrained by explicit policy enforcement.

Threat narrative

Attacker objective: The attacker wants to turn the model’s own privileges, helpfulness, and integrations into a path for unauthorized actions or data exposure.

1. Entry occurs when a malicious prompt, document, or external data source is ingested into an LLM context that also has tool or data access.
2. Credentialed action follows when the model uses its granted permissions to query systems, call APIs, or execute code on the attacker’s behalf.
3. Impact appears as data loss, unauthorized access, harmful content amplification, or manipulated system behavior that looks like legitimate model output.

Breaches seen in the wild

• xAI API key leak 2025: An xAI developer leaked an API key on GitHub that reached 60+ private Grok models and stayed live for about two months before revocation.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI safety and AI security are governance problems with different control objectives. Safety is about constraining what a model outputs; security is about constraining what an adversary can make the system do. The article is right to separate them because one set of controls cannot compensate for the other when the model has tools, memory, and external data access. The practitioner conclusion is to design two test plans, two risk registers, and two ownership models.

Tool-enabled AI creates a machine identity problem, not just an LLM problem. The moment an agent can call APIs or write to a database, it carries privileges that must be issued, scoped, monitored, and revoked. That is why identity governance, secret management, and least privilege become core AI controls, not supporting controls. The practitioner conclusion is to treat agent credentials and permissions as first-class assets in your governance model.

Indirect prompt injection is the clearest example of trust boundary collapse in AI systems. External content is entering the same execution context as trusted instructions, which breaks the assumption that the model can reliably distinguish data from command. That is a design failure, not just a tuning issue. The practitioner conclusion is to hard-separate untrusted content from action authority and audit every path that crosses that boundary.

AI infrastructure exposure is becoming part of the broader credential-leak problem. The article’s reference to exposed AI servers and leaked keys shows that the surrounding stack is often easier to compromise than the model itself. This matters for NHI because API keys, service accounts, and agent tokens are the real control surface in many deployments. The practitioner conclusion is to extend secrets governance and inventory control to AI pipelines, connectors, and agent runtimes.

OWASP-NHI and AI governance now overlap in operational practice. When an AI system authenticates to databases, cloud services, or internal APIs, its privileges, secret handling, and offboarding should be governed like any other non-human identity. That intersection is where many teams are weakest because they still treat the model as a product feature rather than an access-bearing runtime actor. The practitioner conclusion is to align AI deployment controls with identity governance before scaling production use.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Identity Security Buyer's Guide

What this signals

AI security teams need a control model that starts at issuance time, not only at output review. Once an AI system can act through tools, the decisive question is which credentials it holds and how narrowly they are scoped. That is why AI governance is converging with identity governance, especially where service accounts and agent tokens touch production systems.

Credential leakage around AI systems is now a programme-level risk, not an isolated incident class. AI-related credential leaks surged 81.5% year-over-year in 2025, according to the State of Secrets Sprawl 2026, which means the surrounding stack is outpacing the core model as a source of exposure. Teams should assume the attack surface is larger than the chatbot UI and manage the full runtime estate accordingly.

AI boundary failures are becoming a repeatable pattern rather than a one-off anomaly. The governance gap is not whether a model can be helpful, but whether helpfulness is allowed to cross into action authority. Practitioners should plan for untrusted content, delegated access, and autonomous tool use as a combined risk profile, then align controls to that profile before production scale-up.

For practitioners

• Separate safety and security testing programs Run model-behavior tests for harmful output and security tests for prompt injection, data exfiltration, and unauthorized tool use as distinct control families.
• Constrain agent tool privileges Scope each agent to the minimum database, API, and file-system permissions needed for its task, and treat those permissions as revocable credentials.
• Sanitize and label external content Classify incoming emails, files, calendar events, and web content as untrusted inputs before they enter model context or tool execution paths.
• Inventory AI credentials and connectors Track every API key, service account, token, and MCP connection used by the AI stack so revocation and offboarding are operationally possible.
• Log model actions and irreversible events Record tool calls, permission changes, and high-risk outputs in immutable logs so security teams can reconstruct AI-driven incidents quickly.

Key takeaways

• AI safety and AI security solve different problems, so one control stack cannot cover both failure modes.
• LLM deployments become materially riskier once models can use tools, credentials, and external content in the same workflow.
• Identity governance now extends into AI runtimes because agent tokens, API keys, and service accounts are the real control surface.

Key terms

• AI Safety: AI safety is the discipline of preventing an AI system from taking unintended or harmful actions on its own. It focuses on the behaviour the system generates, even when no external attacker is involved. For identity teams, safety is about limiting what the agent can do once it is already operating.
• AI Security Platform: An AI security platform governs how people and agents use AI systems across prompts, responses, files, and tool calls. It goes beyond traditional content filtering by adding intent-aware policy, runtime enforcement, and audit linkage so the organisation can control both the conversation and the action that follows.
• Indirect Prompt Injection: Indirect prompt injection is an attack where malicious instructions are hidden inside content that an AI system reads later. The model may treat that content as context rather than as hostile input, which can influence tool use, data access, or workflow actions if controls are weak.
• Non-Human Identity (NHI): A digital identity assigned to a non-human entity such as a software application, service account, API key, bot, machine, or AI agent that enables it to authenticate and interact with systems without direct human involvement. NHIs now outnumber human identities in most enterprises by 25 to 50 times.

Deepen your knowledge

The NHI Foundation Level course covers NHI governance, agentic AI identity, machine identity security, and secrets management in the industry's only accredited NHI security programme. It helps practitioners connect identity controls to the broader security decisions that AI systems now force into production.