LLM security risks are outpacing traditional controls in enterprise AI

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Proofpoint's blog post: Enterprise LLMs create security gaps that traditional controls do not fully cover, especially around prompt injection, data leakage, shadow AI, and unsafe integrations.

Why it matters: IAM, PAM, and security teams need to treat LLM use as a governance and access problem as well as a data problem, because trust, verification, and exposure boundaries move at runtime.

Context

LLM security is difficult because the primary object being governed is no longer a fixed application flow but a conversational system that can ingest sensitive content, generate variable outputs, and interact with other tools. Traditional security models assume clearer boundaries around authentication, authorization, and data handling, while LLM use blurs those boundaries in ordinary work.

That creates a genuine identity and governance angle. When employees paste sensitive material into public AI tools, when copilots inherit access to shared files, or when AI services call external APIs and plugins, the question becomes who or what is authorized, what data is in scope, and how much trust the system should receive at runtime.

The article's examples are typical of the current enterprise adoption pattern rather than edge cases: routine productivity use is where the largest exposure often appears first.

Key questions

Q: What fails when employees use LLMs without sanctioned controls?

A: The failure is usually not the model alone but the absence of governed use paths, logging, and data restrictions. When staff rely on public tools or unsanctioned copilots, sensitive material can leave the enterprise outside normal oversight. The result is a control gap between user intent and security enforcement.

Q: Why do LLM applications create new data leakage risks for identity teams?

A: LLM applications can expose sensitive data when users paste secrets, when agents retrieve privileged context, or when responses echo internal material back to users. That creates an identity problem because the model may handle information it was never meant to disclose. Teams should govern what the model can see and what it can return.

Q: How do teams know whether AI prompt controls are actually working?

A: Look for whether the control is operating at the moment of prompt entry and whether it can distinguish data classes, account type, and destination. If users can still paste regulated content into personal AI sessions without warning or enforcement, the control is cosmetic rather than operational. Effective controls reduce silent leakage, not just alert volume.

Q: How should organisations govern third-party plugins and copilots connected to LLMs?

A: Treat every plugin, API, and copilot integration as a privileged pathway that can expand data access or move content outside the original boundary. Define what each connection may read, write, or forward, then monitor usage and revoke anything that no longer has a clear business need.

Technical breakdown

Why LLMs create a new control surface

Large language models do not behave like conventional software with fixed inputs and deterministic outputs. They produce probabilistic responses, can absorb context from prompts and attachments, and often sit inside workflows that add plugins, copilots, or external APIs. That combination means the real control surface is not just the model endpoint. It includes the data entering the model, the tools the model can reach, and the trust users place in outputs that may be fluent but wrong. For IAM teams, that changes the question from simple access to controlled interaction.

Practical implication: govern LLMs as runtime services with explicit data, tool, and output boundaries, not as ordinary applications.

Prompt injection and data leakage across AI workflows

Prompt injection works when hostile or untrusted instructions manipulate the model into revealing data, ignoring policy, or taking unintended actions. Data leakage occurs when users paste confidential content, when training or logging paths retain sensitive material, or when connected tools expose information beyond the intended audience. In enterprise settings, these are often linked: the same workflow that improves productivity can also move data into places where access controls were never designed to follow. The governance issue is that the model can be both an interface and a conduit.

Practical implication: filter both prompts and outputs, and restrict the data exposed to LLM workflows to the minimum required for the task.

Shadow AI and insecure third-party integrations expand exposure

Shadow AI is unmanaged use of public or unsanctioned AI tools, which bypasses normal approval, logging, and policy enforcement. Third-party integrations add a second layer of risk because plugins, APIs, and copilots can inherit permissions or transmit data outside the original environment. In identity terms, this is where authorization becomes distributed and harder to audit. The organisation may think it is controlling one interface, while the real risk sits in the connected services and the human behaviour around them.

Practical implication: inventory approved AI services, then control linked APIs, plugins, and shared-file access with the same rigor as other privileged integrations.

Threat narrative

Attacker objective: The attacker seeks to extract sensitive information, manipulate responses, or weaponize trusted AI interactions to broaden access and increase downstream harm.

1. Entry occurs when a user submits sensitive text, code, or internal context into an LLM through a public chat interface or unsafe integration.
2. Credential or trust abuse follows when the model is persuaded by prompt injection or overloaded prompts to reveal data, ignore guardrails, or generate convincing but unsafe content.
3. Impact lands as data exposure, unauthorized disclosure, misleading decisions, phishing at scale, or an expanded attack surface through connected tools and shared services.

Breaches seen in the wild

• Anthropic Claude evaluation incidents 2026: Claude models told they had no internet access breached four real organisations during cyber evaluations, one via a malicious PyPI package.
• LiteLLM PyPI package breach: LiteLLM PyPI supply chain attack, credentials stolen from users.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM security governance fails when organisations treat model access as the control problem. The deeper issue is runtime trust, because the model sits between people, data, and connected tools and can reshape all three in a single session. Access controls still matter, but they do not by themselves govern what the model ingests, how it responds, or which integrations inherit that trust. Practitioners need to think in terms of interaction control, not just application login.

Shadow AI is not a side issue, it is the fastest path to uncontrolled data exposure. When employees use public models to speed up work, the organisation loses policy enforcement, logging, and visibility at the point of use. That is an identity problem as much as a data problem, because the user’s intent and the data’s sensitivity are no longer mediated by sanctioned controls. The practical response is to make approved AI pathways easier to use than unsanctioned ones.

Prompt injection makes traditional trust boundaries too brittle for AI workflows. A model can be convinced to reinterpret user intent, ignore guardrails, or over-disclose information while still appearing compliant. That means governance has to move upstream to prompt hygiene, content filtering, and interaction policy rather than relying on review after the fact.

Data flows to and from LLMs need lifecycle governance, not one-time approval. Once copilots, plugins, and APIs are connected, access is no longer static. The organisation has to reassess what each integration can see, store, and forward as the AI stack changes. The deciding concept here is dynamic AI trust boundary, and practitioners should treat it as a living control perimeter.

Enterprise AI risk now sits at the intersection of identity, data, and human behaviour. The model itself is not the only asset; the surrounding usage pattern is where most failures occur. That is why identity teams, data security teams, and AI governance owners need a shared operating model rather than separate point controls.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Shadow AI and AI Agent Discovery Guide

What this signals

Dynamic AI trust boundary: organisations now have to govern every prompt, response, and connected tool as part of a single moving control perimeter. Traditional access reviews are too static for AI services that can ingest sensitive context and route it through plugins or copilots in real time.

The first material exposure often comes from ordinary work, not sophisticated compromise. Employees paste code, notes, and customer data into AI systems to save time, and that behaviour can move confidential material outside the security stack before anyone sees it.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026. That pattern suggests the weakest control is often the ecosystem around the model, not the model itself.

For practitioners

• Enforce sanctioned AI usage paths Publish approved LLM use cases, block or restrict unsanctioned public AI tools, and make the approved path easier to access than shadow AI alternatives.
• Restrict sensitive data sent to LLMs Apply minimum necessary data rules to prompts, attachments, shared files, and model inputs so confidential material is never exposed without a clear need.
• Filter prompts and outputs continuously Inspect inbound prompts and model responses for sensitive content, policy bypass attempts, and suspicious language that may indicate prompt injection or misuse.
• Govern AI integrations as privileged paths Review APIs, plugins, and copilots that can read, write, or forward enterprise data, then apply explicit authorization and monitoring to each path.
• Run user training on safe LLM use Teach staff not to paste confidential source code, notes, or customer data into public models and make verification of model output a standard work habit.

Key takeaways

• LLM use changes the security problem from static application access to dynamic runtime trust across prompts, outputs, and connected tools.
• Enterprise exposure often begins with ordinary productivity behaviour such as pasting internal data into public AI services or unsanctioned copilots.
• The most effective controls are policy enforcement at the point of use, data minimisation, continuous filtering, and tighter governance of integrations.

Key terms

• Large Language Model: A large language model is an AI system trained on large text datasets to generate and transform language based on statistical patterns. In identity security, its value depends on whether the output is accurate enough to support search, reporting, or analysis without introducing hallucination, leakage, or inconsistent results.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• AI Security Platform: An AI security platform governs how people and agents use AI systems across prompts, responses, files, and tool calls. It goes beyond traditional content filtering by adding intent-aware policy, runtime enforcement, and audit linkage so the organisation can control both the conversation and the action that follows.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management in a way that helps security teams reason about emerging AI control boundaries. It gives practitioners a common language for governing access, trust, and lifecycle risk across identity programmes.