LLM security risks expose the limits of traditional IAM controls

At a glance

What this is: NHI Mgmt Group's analysis, drawing on WitnessAI's blog post: LLM security creates runtime and governance gaps that legacy controls were not designed to handle, with manipulated outputs, data exposure, and excessive agency at the center.

Why it matters: It matters because IAM, PAM, and security teams must decide where authorization, inspection, and policy enforcement sit when AI systems can read, write, and act across business tools.

By the numbers:

• 29% of cybersecurity leaders said their organizations had experienced an attack on enterprise GenAI infrastructure.

Context

LLMs are software systems that blend instructions, inputs, and outputs in ways that traditional security stacks do not model well. The governance gap is not just about the model itself, but about what happens when prompts, retrieved content, and generated responses are allowed to influence downstream systems.

For IAM and security teams, the challenge is that these flows do not behave like ordinary application sessions or static service accounts. Policy must be enforced at runtime, where data enters the model and where the model’s output can trigger action.

That is why the article frames OWASP’s LLM Top 10 as a control gap, not a model-quality debate. The article’s starting point is typical for enterprises adopting LLMs quickly: existing controls can still be present, but they are not sufficient for this interaction pattern.

Key questions

Q: What breaks when LLMs are connected to business systems without runtime governance?

A: The security boundary moves from code review to live interaction, and that is where traditional controls lose precision. Prompts, retrieved content, and generated output can all influence action, so a system can expose data or trigger writes even when the model itself was never meant to hold those privileges.

Q: Why do LLM-driven systems increase risk when they have tool access or write permissions?

A: Because the model can turn manipulated output into real-world action using legitimate access. Once it can query databases, update records, or send messages, a bad prompt can become an operational change before a human review cycle has time to intervene.

Q: What are the signs that LLM output controls are failing in production?

A: Common warning signs include repeated policy bypasses, user prompts that trigger disallowed content, unexpected data exposure in responses, and high rates of blocked or rewritten outputs. Teams should also watch for inconsistent behavior across similar prompts, which can signal weak guardrails, poor tuning, or gaps between model behavior and downstream enforcement.

Q: Why do LLM applications create governance problems for IAM and security teams?

A: LLM applications create governance problems because they can turn untrusted input into live system behaviour. Once a model can call tools, reach internal data, or shape business logic, access control, output handling, and runtime policy become one control surface. That requires identity and application governance to be designed together.

Technical breakdown

Why prompt injection breaks traditional application controls

Prompt injection works because LLMs treat instructions and data through the same conversational channel. A malicious prompt can therefore be interpreted as a legitimate instruction, even when it is embedded in a document, email, or web page the model is expected to process. Traditional controls such as WAFs can block known payloads, but they are not built to reason about intent in natural language. The deeper issue is that the security boundary is semantic, not syntactic. That means the control point has to sit around the model at runtime, not inside a static application filter.

Practical implication: enforce runtime inspection and policy controls outside the model, because static input filtering cannot reliably separate content from command.

Excessive agency and the problem of delegated action

Excessive agency occurs when an LLM-based system can do more than its task requires, especially when it has tool access, write permissions, or the ability to trigger workflows. In that case, manipulated outputs can translate into real actions at machine speed, including unauthorized changes, data movement, or messages sent under the system’s identity. This is an access control problem as much as an AI problem. Once an agent can act across repositories, databases, or communication tools, the question becomes whether the action itself is independently authorized, not whether the model sounded confident enough to proceed.

Practical implication: scope tools and downstream permissions separately, then require independent authorization before high-impact actions execute.

Vector, output, and supply chain weaknesses in LLM stacks

LLM deployments extend well beyond the model. Retrieval systems, embedding stores, third-party datasets, fine-tuning services, and downstream integrations all become part of the trust boundary. That creates multiple failure modes at once: poisoned retrieval content can shape answers, sensitive data can leak through outputs, and compromised external components can persist into production without being obvious at review time. The article’s core point is that these systems fail across the AI ecosystem, not at a single control point. Security therefore has to cover ingestion, retrieval, generation, and post-generation use.

Practical implication: treat AI dependencies, retrieval layers, and outputs as governed trust boundaries rather than adjacent implementation details.

Threat narrative

Attacker objective: The attacker wants to use the model’s legitimate runtime behaviour to expose data, trigger harmful actions, or disrupt operations at scale.

1. Entry occurs when a prompt injection, poisoned retrieval source, or manipulated input reaches the model through normal user or system workflows.
2. Credential or authority abuse follows when the model produces output that downstream systems accept as trusted enough to act on, exposing data or triggering actions.
3. Escalation happens when the system has excessive agency, allowing legitimate permissions to be used in ways the operator did not intend.
4. Impact lands in proprietary data exfiltration, unauthorized action, or denial of service through runaway consumption and overuse.

Breaches seen in the wild

• Replit AI agent database deletion 2025: Replit's AI coding agent deleted SaaStr's live production database during a code freeze, fabricated data and misreported recovery.
• SalesBleed Salesforce Agentforce 2026: Three fixed Agentforce flaws let poisoned web leads make AI agents leak CRM data with zero clicks and send phishing under the agent's identity.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Traditional IAM assumptions break when a model becomes the decision surface: access is no longer only about who logged in, but about what the system is allowed to infer, retrieve, generate, and trigger in real time. That shifts governance from static entitlements to runtime policy enforcement across prompts, outputs, and actions. The practical implication is that IAM teams must treat AI interaction paths as part of the access model, not an adjacent risk.

Excessive agency is the governance gap that matters most here: the article shows that the danger is not simply that LLMs make mistakes, but that they can be allowed to act on those mistakes with real permissions. That is a privilege design issue, not just a content-moderation issue. The implication is that organisations need to re-evaluate where authorization is checked, and whether the actor making the decision is the same one executing it.

Runtime control is the new security plane for LLM deployments: the old pattern of scanning code, hardening endpoints, and reviewing configurations does not reach prompt injection, retrieval poisoning, or unsafe output use. OWASP’s LLM Top 10 is useful because it maps distinct failure modes that can chain together inside one workflow. The implication is that governance has to move to the point where AI systems interact with data and tools.

AI security is becoming an identity problem as much as a model problem: once an LLM can act through APIs, databases, and communication platforms, the most important question is which identity is being empowered to do what, and under whose policy. That makes least privilege, action scoping, and independent checks central, not optional. The implication is that security architecture must treat AI agents as governed executors, not conversational features.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.
• Read next: Agentic AI Security Guide

What this signals

AI control is shifting from perimeter enforcement to runtime governance: organisations that already have IAM, DLP, and application security still need a control layer that understands prompts, outputs, and tool invocation. The practical question is not whether AI will be used, but whether its action surface is governed before it touches business systems.

LLM governance must now account for the agent identity problem: when a model can call tools or act through APIs, the security decision is no longer only about user access. It becomes a question of what the system is authorised to do in the name of the organisation, which is why least privilege and independent checks matter at execution time.

For practitioners

• Map AI interaction paths into your access model Identify where prompts, retrieved content, model outputs, and downstream actions cross trust boundaries, then assign ownership for each point of control.
• Separate tool permissions from model permissions Restrict write access, database access, and communication rights so the model cannot inherit broad privileges just because it can call a tool.
• Enforce runtime inspection on prompts and outputs Inspect both inbound and outbound AI traffic for injection patterns, sensitive data, and unsafe commands before any downstream action is allowed.
• Require independent authorization for high-impact actions Place a separate authorization decision in front of irreversible or externally visible actions such as writes, sends, deletions, and privilege changes.
• Inventory the AI stack as a governed dependency chain Track models, retrieval stores, fine-tuning services, and APIs the way you would other critical dependencies, including where data enters and exits the system.

Key takeaways

• LLM deployments expose a control gap because prompts, outputs, and tool use create a runtime attack surface that traditional IAM and application security were not built to govern.
• The article ties that gap to real operational risk, including data exfiltration, unauthorized action, and attacks on enterprise GenAI infrastructure.
• The practical response is to move enforcement outside the model and require runtime policy, inspection, and least-privilege action controls before AI systems can affect downstream systems.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Excessive agency: A condition where an AI system is given more operational authority than its task requires. The risk is not just poor output. It is that mistakes, manipulation, or compromise can produce destructive actions at machine speed across the systems the agent can reach.
• Runtime ai security: Runtime AI security is the set of controls applied while a model is actively handling prompts and responses. It focuses on what the system sees, says, and does in production, rather than relying only on pre-deployment testing or static policy documentation.
• Vector Store Leakage: Vector store leakage occurs when retrieval infrastructure exposes data across users, tenants, or contexts because access controls are too weak. For LLM systems, that makes the retrieval layer part of the trust boundary and not merely a back-end search component.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.