LLM security gaps show why traditional controls miss AI threats

At a glance

What this is: This is a Panther guide to LLM security, showing that traditional application controls do not map cleanly to prompt injection, data leakage, model poisoning, extraction, and supply chain abuse.

Why it matters: It matters because teams now need governance for AI inputs, outputs, tools, and retrieval paths, not just perimeter controls, if they want to prevent LLMs from becoming an attack surface.

By the numbers:

• 13% of organisations reported breaches involving AI models or applications in 2024.
• 97% of those organisations lacked proper AI access controls.

Context

LLM security is the discipline of protecting language models, the data they process, and the systems they can reach through prompts, tools, APIs, and retrieval layers. The control gap is structural: these systems do not behave like conventional applications, so standard assumptions about trust boundaries, signatures, and validation often fail.

For identity and security teams, the important point is that LLMs increasingly sit inside production workflows with access to data and actions, which creates governance questions around permissions, outputs, and delegated use. The article’s framing is typical of the current market reality: organisations are adopting capabilities faster than they are designing controls around them.

Key questions

Q: What fails when LLMs are protected only with traditional application security controls?

A: Traditional controls fail because LLMs collapse trusted and untrusted text into the same context window, so signatures, protocol checks, and ordinary DLP rules often miss prompt injection, retrieval poisoning, and model-driven abuse. Security has to move from static inspection to context-aware governance of prompts, outputs, tools, and retrieval sources.

Q: Why do LLMs increase data exposure risk when employees use them for everyday work?

A: LLMs increase exposure risk because users often paste sensitive business data into prompts, connectors can pull data from approved systems, and outputs can repeat confidential material. The risk is not only leakage in the prompt itself, but also in retrieval, model responses, and downstream storage. Without policy and inspection, ordinary productivity use can become an uncontrolled data path.

Q: What are the signs that an LLM’s safety controls are failing in practice?

A: Common warning signs include toxic or biased responses, inconsistent refusal behavior, over-acceptance of harmful prompts, and weak handling of demographic-sensitive content. If the model appears helpful but cannot reliably distinguish safe from unsafe output, the control layer is too loose. Teams should look for these symptoms across evaluation sets, not only in isolated demonstrations or benchmark scores.

Q: How should security teams govern LLMs that can call tools or run code?

A: Security teams should govern them as privileged workloads, not as chat interfaces. Every tool call needs server-side authorization, schema validation, and a dedicated identity with minimal scope. If a model can influence code execution, treat that path like a production admin channel and apply least privilege, logging, and runtime isolation.

Technical breakdown

Why traditional application controls miss LLM attack paths

Traditional controls are built around predictable protocols, bounded inputs, and deterministic behaviour. LLMs break those assumptions because the same text channel carries both trusted instructions and untrusted user content, and the model decides how to interpret it at runtime. That means SQL-style signatures, web filters, and normal DLP rules often miss attacks that live entirely inside prompt content. In practice, the security problem is not just malicious input. It is the combination of natural-language flexibility, opaque model internals, and downstream tool access that turns a model response into an execution path.

Practical implication: Treat LLM prompts, retrieval content, and outputs as security-relevant control points, not as ordinary application text.

How prompt injection and RAG poisoning change the threat model

Prompt injection overrides intended behaviour by embedding instructions that the model treats as part of the conversation. Indirect injection is especially dangerous in retrieval-augmented generation, because malicious content can enter through documents, web pages, or knowledge bases before being surfaced in a prompt. RAG poisoning goes one step further by corrupting the retrieved context itself, so the model amplifies attacker-supplied material as if it were trusted evidence. These are governance failures as much as technical failures, because the model is only as safe as the content pipeline around it.

Practical implication: Sanitise retrieved content and validate sources before it reaches the model context window.

Why tool access turns LLMs into privilege-bearing systems

Once an LLM can call APIs, query databases, or trigger workflows, the model stops being a passive text generator and becomes a privilege-bearing system. That is where identity and access governance matters most, because the real question is no longer what the model says but what it can do on behalf of a user or workflow. Overbroad tool permissions, weak output validation, and poorly constrained agent actions can convert a simple prompt into data exposure or unauthorised execution. This is why agent access should be governed like any other high-risk non-human identity.

Practical implication: Scope model tool permissions to the minimum required actions and enforce approval boundaries for any write or destructive operation.

Threat narrative

Attacker objective: The attacker aims to steer the model into leaking data, executing unauthorised actions, or providing a trusted channel for malicious activity without triggering conventional detections.

1. Entry occurs through prompt injection, indirect prompt injection, or malicious content inside retrieved knowledge that the LLM later consumes as context.
2. Credential or access abuse follows when exposed cloud credentials, API access, or permissive tool bindings let the model reach systems it should not control.
3. Impact comes from data leakage, unauthorised actions, model theft, or use of legitimate AI services as covert infrastructure and command-and-control channels.

Breaches seen in the wild

• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM security is now an identity and delegation problem, not just a content-safety problem. Once a model can query tools, retrieve data, or trigger workflows, the core risk becomes who or what is allowed to act through it. That makes access governance, approval boundaries, and identity-bound permissions central to the control model. Practitioners should treat model-mediated action as a governed delegation chain, not as plain application output.

Prompt injection reveals a control boundary failure, not a simple filtering problem. The article shows why text-based allowlists and legacy signatures are insufficient when the attack and the legitimate request look identical at the protocol level. Security teams need controls that evaluate context, source trust, and downstream action risk. The practical conclusion is that prevention and detection must move closer to issuance, retrieval, and execution time.

LLM governance debt: is the growing gap between where AI systems are deployed and where controls actually exist. The more LLMs are wired into APIs, databases, and agents, the more expensive it becomes to bolt on after-the-fact controls. This is especially visible in environments that assumed existing IAM and DLP policies would automatically cover model behaviour. Teams should assume that AI governance must be designed into the operating model, not retrofitted after incidents.

AI agents extend the NHI problem into a new trust layer. The article’s discussion of agents with tool access shows that machine identities now include systems that make runtime decisions, not just service accounts and tokens. That changes inventory, approval, and monitoring requirements because the risky entity is the agentic system itself. Security leaders should classify agent identities separately from ordinary automation and govern them accordingly.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

LLM governance debt: the longer organisations rely on conventional application controls, the more AI risk accumulates outside existing IAM, DLP, and monitoring models. That debt shows up when teams realise the model’s real attack surface is its prompts, retrieval sources, and tool bindings, not just the application wrapper.

The control shift is toward issuance-time and execution-time governance, because prompt filtering alone will not stop attacks that arrive through retrieved content or delegated actions. Security teams should assume that AI agents will be treated as governed non-human identities, with permissions and monitoring designed around what they can do rather than what they can say.

For practitioners

• Enforce least-privilege tool access for LLMs Limit each model or agent to the smallest set of read and write actions it genuinely needs, and separate retrieval, execution, and administration permissions.
• Validate retrieved content before prompt injection Inspect RAG sources, knowledge base entries, and external content for malicious instructions before they enter the model context window.
• Treat model outputs as untrusted data Run every LLM response through the same validation logic you would apply to user input before sending it to a database, shell, or downstream API.
• Log prompts, outputs, and tool calls together Capture the full prompt-response pair, retrieved context, and tool invocation trail so investigations can reconstruct what the model saw and did.

Key takeaways

• LLM security problems arise because natural-language systems do not preserve the same trust boundaries that conventional applications rely on.
• The evidence in the article shows that organisations are already suffering model and application breaches while lacking proper AI access controls.
• Practitioners need governance over prompts, retrieval content, tool permissions, and outputs, because the dangerous action often happens after the model has been prompted.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Retrieval-augmented Generation: Retrieval-augmented generation is a pattern where an AI model pulls external information before generating output. The security challenge is that access rules can weaken when data is chunked, embedded, cached, or reused, so source permissions may not automatically follow the content into the model's context.
• Model Extraction: Model extraction is the theft of model behaviour or parameters through repeated interactions, often via an exposed API. The attacker may not need direct file access if the interface leaks enough information through outputs, rate limits, or weak request controls.
• Tool-Bearing Agent: A tool-bearing agent is an AI system that can decide when to call external services, query data, or trigger actions beyond generating text. That makes it a governed non-human identity with real access scope, audit needs, and potential privilege abuse paths.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance and machine identity security in terms security and identity teams can apply to delegated AI systems. It is designed for practitioners who need a common control language for access, approval, and lifecycle governance across modern identity programmes.