LLM security risks are outpacing traditional controls in enterprise AI

At a glance

What this is: This is Proofpoint’s analysis of why LLMs create a distinct security problem, centered on data leakage, prompt injection, shadow AI, and risky integrations.

Why it matters: It matters because LLMs are increasingly handling sensitive work, which means IAM, data protection, and usage governance must account for model behaviour, not just user access.

👉 Read Proofpoint’s analysis of LLM security risks and enterprise controls

Context

Large language models have moved from experimental tooling to operational systems that touch customer service, coding, analysis, and internal decision-making. That shift creates a security and governance gap because the model can process sensitive inputs, generate confident but incorrect outputs, and interact with other systems in ways traditional application controls were not designed to manage.

For identity and access teams, the relevant issue is not only who can log in, but what the model can see, retain, infer, and trigger once access is granted. In LLM environments, access control, data minimisation, and verification controls need to work together because the model itself becomes part of the trust boundary.

Key questions

Q: What breaks when employees use public LLM tools with confidential data?

A: When employees enter confidential data into public LLM tools, the organisation loses control over retention, reuse, and downstream exposure. That creates a governance problem, not just a privacy one, because the organisation may no longer know where the data was processed or how the output could be reused. Approved tools, policy enforcement, and user education need to close that gap.

Q: Why do LLMs create risk in identity and access management?

A: LLMs create risk when teams confuse fluent answers with verified security evidence. A model can summarize access patterns quickly, but it can also hide missing context, stale data, or inaccurate scope. In IAM, that means every output needs traceability back to the underlying identities, entitlements, and events.

Q: What do security teams get wrong about prompt injection defence?

A: They often assume better blocklists will solve the problem, but obfuscation simply changes the shape of the payload. Real defence requires examining meaning across the full interaction, including retrieved content and model responses. If the control cannot interpret intent, it will keep missing the attack class it is meant to stop.

Q: How should security teams govern shadow AI without blocking business productivity?

A: Start by identifying the identities and credentials behind AI use, then classify each one by data sensitivity, connected systems, and business purpose. Governance works best when organisations control the access path rather than banning the tool outright. That means inventory, approval, monitoring, and revocation all need to follow the same identity path.

Technical breakdown

Why LLMs create a different security model

Large language models do not behave like deterministic applications. They produce probabilistic outputs based on prompts, training data, and surrounding context, which means the same input can yield different results over time. That unpredictability creates governance difficulty because security teams cannot rely on fixed rules alone to infer safe behaviour. LLMs also interact with external tools, APIs, and plugins, which widens the attack surface beyond the model itself. Once a model can read data, call services, or return generated content to users, the security problem becomes one of controlling inputs, outputs, and delegated actions together.

Practical implication: Treat LLMs as controlled systems with data and action boundaries, not as ordinary software components.

Prompt injection, data leakage, and model manipulation

Prompt injection is a manipulation technique where a user or adversary crafts input to override intended instructions, steer model behaviour, or expose hidden context. Data leakage can occur when the model reveals sensitive material in prompts, responses, logs, or downstream integrations. Model inversion is a related risk in which an attacker tries to infer training data from model behaviour. These are not classic vulnerabilities in the code sense. They are interaction-level failures that emerge when the model is trusted to interpret context that should have been filtered, constrained, or separated from sensitive content.

Practical implication: Place filtering, classification, and redaction controls in front of model inputs and outputs, especially where confidential data is involved.

Shadow AI and insecure third-party integrations

Shadow AI appears when employees use public or unapproved LLM tools without oversight, often by entering work data into systems outside approved governance. Insecure third-party integrations create a similar problem because plugins, APIs, and connected services can extend trust far beyond the model owner’s direct control. The risk is not only data exposure. It is also the creation of hidden dependency chains that bypass approval, logging, and policy enforcement. For IAM and security leaders, this makes usage visibility and entitlement control as important in AI environments as they are in SaaS and cloud ecosystems.

Practical implication: Inventory approved models, connectors, and user populations so ungoverned AI usage can be detected and constrained.

Threat narrative

Attacker objective: The attacker aims to extract sensitive information, influence outputs, or use the model and its integrations as a pathway to wider data exposure and operational harm.

1. Entry begins when users paste confidential material into public LLM tools or when an attacker crafts a prompt that manipulates the model’s behaviour.
2. Escalation occurs when the model reveals hidden context, interacts with connected tools, or follows malicious instructions that bypass intended guardrails.
3. Impact follows as sensitive data is exposed, incorrect decisions are made, or unapproved AI use creates compliance and trust failures.

NHI Mgmt Group analysis

LLM security is now an access-governance problem, not only a model-risk problem. The article shows that the most damaging failures come from how people and systems interact with the model, especially when confidential data, plugins, or unapproved tools are involved. That means identity, access, and data controls have to be aligned around the model’s real trust boundary. Practitioners should treat LLMs as governed access endpoints, not as passive productivity tools.

Shadow AI creates the same governance blind spot that unmanaged SaaS once created, but with faster consequences. When employees use public LLMs outside policy, the organisation loses visibility into what data was entered, which identities were involved, and whether the output was retained or reused. The named concept here is LLM trust boundary drift: the point at which model usage expands beyond the controls designed to protect it. Security teams should assume that unapproved AI use will continue unless discovery and policy enforcement are built into the workflow.

Prompt injection shows that input validation alone is not enough when the system can take actions on behalf of users. The issue is not only malicious text, but the model’s ability to convert that text into harmful downstream behaviour through APIs, plugins, or connected services. This is where AI governance and IAM intersect: delegated actions need the same scrutiny as any other privileged workflow. Practitioners should redesign authorization around constrained actions, not just authenticated sessions.

Over-reliance on model output is already a business control issue, not a future concern. The article highlights that LLMs can sound convincing while still being wrong, biased, or incomplete, which means human reviewers may defer to outputs they should challenge. For enterprises, that creates audit, compliance, and operational risk in equal measure. Security leaders should require verification paths for high-impact decisions and ensure model use is paired with accountable human review.

The security model for LLMs must include data minimisation, auditing, and scoped third-party access. The article’s best-practice section points to the right direction, but the deeper lesson is that governance breaks when the model can see more than it needs or call more than it should. That makes least privilege, logging, and policy enforcement central to AI security programmes. Practitioners should make the model’s access scope auditable and enforceable.

What this signals

LLM trust boundary drift: the control problem emerges when model use expands beyond the identities, data sets, and workflows originally approved. That drift is already visible in agent behaviour, where autonomy and tool use can exceed intended scope in ways traditional review processes miss. Teams that do not map LLM access to identity, data, and connector boundaries will struggle to explain where sensitive information went or who approved the action.

The practical response is to make AI usage observable at the same level as SaaS and cloud entitlements. Discovery, audit logging, and policy enforcement need to cover prompts, outputs, connectors, and data sources so security, compliance, and identity teams can work from the same evidence set.

As LLMs move deeper into business workflows, governance will shift from model approval to continuous control of model behaviour. That means the next maturity step is not just blocking unapproved tools, but defining what the model is allowed to see, say, and do under OWASP NHI Top 10-style risk thinking.

For practitioners

• Inventory all approved LLM usage Track which models, connectors, and business units are using public or private LLM tools so you can separate sanctioned use from shadow AI. Include browser-based usage, embedded copilots, and API-connected workflows in the same inventory.
• Apply least privilege to model-connected data sources Limit the datasets, shared files, and APIs that LLMs can reach to the minimum required for each workflow. Review permissions for enterprise copilots and connected apps on a recurring basis.
• Filter prompts and responses for sensitive content Use input and output controls to block confidential material before it reaches a model and before model output reaches users. Pair classification rules with redaction and logging so risky interactions can be investigated.
• Constrain third-party integrations and plugins Approve only integrations with documented ownership, logging, and access boundaries. Revoke unused connectors and require authentication and policy checks before any model can act through an external service.
• Require human verification for high-impact outputs Establish review steps for decisions that affect legal, financial, security, or customer outcomes. If the model is drafting or recommending, a trained reviewer should validate the result before it is used operationally.

Key takeaways

• LLMs introduce a governance problem because their outputs are probabilistic, their inputs can contain sensitive data, and their integrations expand the attack surface.
• The operational evidence is already visible in shadow AI, prompt manipulation, and examples of real-world data exposure through ordinary user behaviour.
• The right response is to combine least privilege, filtering, auditability, and human verification so the model’s trust boundary stays enforceable.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• Model Inversion: Model inversion is an attack technique that tries to recover sensitive information from a model by querying it in a targeted way. The risk matters because information hidden in training data, prompts, or internal behaviour can sometimes be inferred from outputs rather than directly accessed.

What's in the full article

Proofpoint's full blog post covers the operational detail this post intentionally leaves for the source:

• Step-by-step guidance for filtering inputs and outputs before they reach public or internal LLMs
• Practical examples of prompt monitoring and AI security gateway controls for enterprise workflows
• Detailed use cases for data loss prevention and access controls around copilots and shared files
• Operational recommendations for identifying and responding to shadow AI across the organisation

👉 Proofpoint’s full post covers the LLM incident examples, control patterns, and recommended safeguards in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and agentic AI identity in practical terms. It helps security and identity practitioners build governance that matches how modern systems actually authenticate, access data, and delegate actions.