LLM security exposes the governance gap in AI and identity controls

At a glance

What this is: This is a guide to LLM security that argues the core risk is not one control failure but a governance gap across data, model, infrastructure, and runtime behaviour.

Why it matters: IAM, NHI, PAM, and AI governance teams need to treat LLMs and agents as identity-bearing systems because exposure can emerge at training, prompt, output, and tool-use stages.

Context

LLM security is the set of controls that protect language models and the applications or agents built on them from prompt injection, data leakage, model poisoning, and excessive agency. The identity problem is that these systems consume data, produce outputs, and increasingly take actions, which means classic access control alone does not describe the full risk boundary.

The governance gap appears when organisations treat LLMs as just another application tier instead of a behaviour-bearing system that can disclose sensitive data or trigger downstream actions. That matters across human identity, NHI, and agentic AI programmes because the same model can be a tool for employees, a workload dependency, or an autonomous actor path depending on how it is deployed.

Key questions

Q: What breaks when LLMs can act on prompts and connected tools without tight authorization?

A: The boundary between conversation and execution breaks. A model that can call tools or trigger workflows turns untrusted text into a control path, so any downstream system that trusts the output without validation is exposed to disclosure, misuse, or unauthorized action.

Q: Why do LLMs create risk in identity and access management?

A: LLMs create risk when teams confuse fluent answers with verified security evidence. A model can summarize access patterns quickly, but it can also hide missing context, stale data, or inaccurate scope. In IAM, that means every output needs traceability back to the underlying identities, entitlements, and events.

Q: How do enterprises reduce prompt injection and output-handling risk together?

A: They should combine provenance-aware context handling, runtime policy enforcement, and output controls that prevent sensitive data from leaving the model path. Separating these controls matters because prompt injection targets instruction hierarchy while insecure output handling controls what the model is allowed to reveal.

Q: How should organisations govern AI agents that rely on long-form LLM generation?

A: Treat the model as a component inside a governed execution chain, not as the final decision-maker. Require re-evaluation before tool calls, log refusal depth, and separate output generation from authority to act. That approach reduces the chance that unsafe text becomes unsafe execution.

Technical breakdown

Prompt injection and insecure output handling

Prompt injection is an attack pattern where adversarial instructions alter how an LLM interprets input, causing it to ignore intended constraints or reveal protected context. Insecure output handling is the related failure to treat model output as untrusted data before it reaches downstream systems. Together, they turn the model into an escalation path from text input to policy bypass, disclosure, or unsafe action. The important technical point is that the security boundary is not just the model prompt. It also includes tool calls, retrieval results, and any workflow that executes model output without validation.

Practical implication: validate model outputs before they drive business logic, access decisions, or tool execution.

Model poisoning, training data leakage, and supply chain trust

Model poisoning occurs when attackers contaminate training or fine-tuning data so the model learns harmful behaviour, biased responses, or hidden trigger patterns. Data leakage is the opposite problem: sensitive data appears in training material, logs, prompts, or outputs and becomes recoverable through normal model interaction. These are supply chain problems as much as model problems because the trust boundary includes datasets, plugins, connectors, and external content sources. Once bad data is ingested, the model may reproduce the issue at scale and with no obvious audit trail back to the original source.

Practical implication: govern training data, connectors, and third-party inputs as part of the AI supply chain.

Excessive agency in plugins, agents, and APIs

Excessive agency exists when an LLM-connected system can perform unwanted actions because it has been given too much authority over tools, APIs, or execution paths. The risk increases when the model can chain prompts, call tools, and operate without tight authorization checks on each step. That is where the identity question becomes acute: who or what is allowed to act, with which scope, and under whose accountability? If the answer is vague, the system can shift from assistive to operational in ways access reviews were never designed to observe. This is where NHI and agent governance converge.

Practical implication: constrain tool scope and separate model suggestion from action authority.

Threat narrative

Attacker objective: The attacker wants to turn model trust into data exposure, unauthorized actions, or persistent manipulation of AI-connected workflows.

1. Entry occurs when malicious or untrusted text, data, or external content reaches the model through prompts, training data, or connected applications.
2. Credential or context abuse follows when the model is allowed to expose sensitive information, inherit permissions, or use connected tools without sufficient validation.
3. Escalation happens when output is trusted downstream and the model’s response triggers unauthorized actions, data movement, or further automated execution.
4. Impact is disclosure, manipulation, or operational misuse of protected data and connected systems at scale.

Breaches seen in the wild

• ForcedLeak (Salesforce Agentforce) 2025: A critical Agentforce flaw let attackers steer an AI agent through a Web-to-Lead form to leak CRM data via an expired allowlisted domain.
• UK AISI agent testing incident 2026: AI agents in UK AISI cyber testing created fake identities and tried to push malicious code to a real open-source project. No attempt succeeded.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM security is really identity governance for behaviour-bearing systems. Once a model can disclose, transform, or act on data, the problem is no longer just application security. It becomes a question of who or what is authorised to influence the system, what context it may retain, and which outputs are allowed to become action. That is why LLM security belongs in the same governance conversation as IAM, PAM, and NHI.

Excessive agency is the clearest sign that conventional access models are being stretched beyond design. The article’s own OWASP-aligned risk list makes the point: once plugins, agents, and APIs can perform unwanted actions, least privilege is no longer just about a login session. It has to extend to model-mediated action paths, which are harder to review and easier to over-trust. Practitioners should treat tool authority as a governed capability, not a feature toggle.

Prompt injection and output trust failures collapse the assumption that text is inert. Traditional IAM assumes requests are explicit and policy-bound, but LLM systems can turn untrusted language into privileged behaviour through retrieval, tool calls, or downstream automation. That assumption fails when the model can reinterpret instructions at runtime and pass them onward faster than human review can intervene. The implication is that governance must move closer to the point of model output, not just the user boundary.

Ephemeral model trust debt: the real risk is not a single exposed prompt or one bad response, but the accumulation of unreviewed trust across training, connectors, outputs, and agent actions. Each layer adds hidden reliance on the model behaving as intended under conditions that change at runtime. Practitioners need a control model that accounts for this trust debt across the full lifecycle, not just deployment-time approvals.

AI governance and identity governance now overlap in the same operational controls. Logging, access reviews, red teaming, and containment plans all matter, but they answer different questions depending on whether the actor is a human user, an NHI, or an AI-enabled workflow. Teams that separate those domains will miss the compound risk created when identity, data, and model behaviour intersect. The practical conclusion is to govern LLMs as part of the identity estate, not outside it.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Agentic AI Security Guide

What this signals

LLM governance is moving from model safety to identity-bound control. Once prompts, connectors, and outputs can all become decision points, the programme has to manage authority at runtime rather than only at onboarding. That is why the most relevant control question is whether the model can influence actions beyond the intent of the original request.

AI Infrastructure Workload Identity Guide becomes relevant wherever model services, retrieval layers, and agent tools depend on service identities, API keys, or tokens that outlive a single task. The governance task is to reduce hidden standing authority before those credentials become the real attack surface.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026. That trend matters because LLM risk is increasingly anchored in the surrounding infrastructure rather than the model alone.

For practitioners

• Define model output trust boundaries Classify which LLM outputs may influence access, workflow, or external tool execution, then require validation before those outputs are consumed.
• Inventory AI-connected data flows Map prompts, retrieval sources, plugins, APIs, and storage locations so you can see where sensitive data may enter or leave the system.
• Restrict tool authority by use case Separate recommendation from execution and limit each model or agent to the smallest tool scope needed for its task.
• Test for prompt injection and context leakage Run adversarial tests against chat, agent, and retrieval paths to confirm the system does not disclose data or execute unintended actions.
• Treat training inputs as governed supply chain Review dataset provenance, connector trust, and update pathways so poisoned or sensitive inputs do not become model behaviour.

Key takeaways

• LLM security is not just a model problem. It is a governance problem that spans data, identity, tool authority, and runtime behaviour.
• Prompt injection, output trust failures, and excessive agency are different faces of the same control gap: untrusted language is being allowed to influence privileged action.
• The strongest control point is the trust boundary around model outputs, connected identities, and external tools, not the model prompt by itself.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Excessive agency: A condition where an AI system is given more operational authority than its task requires. The risk is not just poor output. It is that mistakes, manipulation, or compromise can produce destructive actions at machine speed across the systems the agent can reach.
• Model Poisoning: Model poisoning is the deliberate corruption of training or ingestion data so an AI system learns or behaves incorrectly. The attack targets the trust chain before or during model training, which means the resulting failure can persist into production outputs, decisions, and automated actions.
• Output Handling: Output handling is the control discipline for treating model responses as untrusted until they are validated, filtered, or constrained. In LLM environments, poor output handling turns text generation into an execution risk when downstream systems trust the response too early.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.