LLM security gaps show why traditional controls miss AI threats

At a glance

What this is: This is a Panther analysis of LLM security risks, showing that natural-language models create an attack surface that traditional application, data, and endpoint controls do not reliably cover.

Why it matters: It matters because IAM, PAM, and security teams now have to govern model access, prompt boundaries, tool permissions, and sensitive data flow as part of the broader identity control plane.

By the numbers:

• In 2024, 13% of organizations reported breaches involving AI models or applications, and 97% of those organizations lacked proper AI access controls.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.

👉 Read Panther's guide to LLM security risks, vulnerabilities, and best practices

Context

LLM security is the discipline of protecting large language models, their surrounding infrastructure, and the data they touch from misuse, leakage, and manipulation. The core governance problem is that LLMs do not preserve the same trust boundaries that conventional applications rely on, which makes identity, access, and output validation central to the security model.

This matters to IAM and NHI practitioners because LLM systems increasingly rely on API keys, service accounts, retrieval layers, and tool permissions to operate. Once those credentials or delegated capabilities are exposed, the model becomes part of the attack path rather than just the target, which is why the boundary between application security and identity security is now materially blurred.

The article's starting position is typical of the current market: teams understand the risk in broad terms, but operational control patterns are still immature and unevenly implemented.

Key questions

Q: How should security teams govern LLM and agent access in production?

A: Treat every model, agent, connector, and service account as part of one identity surface. Define owners, scope permissions to task and environment, and require logging that shows who or what accessed the system, what data was touched, and what downstream actions occurred.

Q: Why do LLMs create more risk when they can call tools or APIs?

A: Tool access turns the model from a text generator into a delegated operator. If its permissions are broad, an attacker can steer it into reading data, changing systems, or triggering workflows the organisation never intended. Least privilege and explicit approval boundaries are therefore central controls.

Q: What do security teams get wrong about prompt injection defence?

A: They often assume better blocklists will solve the problem, but obfuscation simply changes the shape of the payload. Real defence requires examining meaning across the full interaction, including retrieved content and model responses. If the control cannot interpret intent, it will keep missing the attack class it is meant to stop.

Q: Which frameworks should practitioners use for LLM security governance?

A: For AI-specific risk, use NIST AI Risk Management Framework and OWASP Agentic AI Top 10. Where credentials and workload access are involved, add OWASP NHI guidance, NIST Cybersecurity Framework, and identity controls that cover secrets, privilege, and auditability.

Technical breakdown

Why prompt injection breaks the trust model in LLM workflows

Prompt injection works because the model processes user instructions, system instructions, and retrieved content as the same token stream. That means malicious text can override intended behaviour without triggering the kinds of protocol validation or signature-based detection used in ordinary application security. Indirect injection is especially difficult because the malicious instruction may arrive through a document, web page, or retrieval system that the model later treats as context. In practice, the attack is not a code exploit in the classic sense. It is a control-plane failure in how the system separates trusted from untrusted language.

Practical implication: restrict what untrusted content can influence, and validate prompt and retrieval flows before they reach downstream actions.

How sensitive data leaks from training data, prompts, and RAG layers

LLMs can expose sensitive data in three different ways. First, training data can be memorised and regurgitated. Second, context contamination can reveal system prompts or internal instructions. Third, retrieval-augmented generation can leak cross-tenant or over-permissioned data if access checks are weak. These failure modes matter because they bypass the assumptions behind DLP and perimeter controls. The model is not just generating text, it is transforming data across boundaries that may not be visible to the security stack. That makes provenance, access control, and output review part of the same governance problem.

Practical implication: enforce access checks on retrieval sources and treat model output as sensitive until it is validated.

Why LLM tool access creates a privilege escalation path

When an LLM can call APIs, run code, or trigger workflows, it no longer functions as a passive interface. It becomes an actor with delegated privileges. If those privileges are broad, the model can be steered into actions that were never intended by the operator, including data exfiltration, system changes, or resource exhaustion. This is where identity governance intersects directly with AI security: the effective privilege of the model is the sum of its tool scope, credential exposure, and the permissions granted to any backing service accounts. Without tight scoping, an LLM can become a proxy for over-privileged access.

Practical implication: map every model tool call to least privilege, short-lived credentials, and explicit approval boundaries.

Threat narrative

Attacker objective: The attacker wants to turn legitimate AI workflows into an access path for data theft, unauthorized execution, or covert control of enterprise systems.

1. Entry occurs when attackers exploit exposed credentials, indirect prompt injection, or compromised AI APIs to reach an LLM workflow.
2. Credential access or abuse follows when the model or its backing services accept delegated secrets, service tokens, or overly broad API permissions.
3. Escalation happens when the attacker uses the model's tool access, retrieval scope, or downstream integrations to reach data and systems beyond the intended boundary.
4. Impact is data leakage, unauthorized actions, model hijacking, or using the AI service for command-and-control and other malicious operations.

NHI Mgmt Group analysis

LLM security is now an identity problem as much as a model problem. The article is right to frame natural-language attacks as something traditional security tools were not designed to see, but the deeper governance issue is privilege, not prose. Once an LLM can read retrieval content, invoke tools, or use API keys, its effective identity must be governed like any other high-risk workload. That means IAM, PAM, and secrets management become core AI controls, not adjacent ones. Practitioners should treat model access as a first-class identity estate.

Prompt injection is really a boundary collapse between trusted instructions and untrusted data. The control failure is not just that attackers can write clever prompts. It is that the system architecture allows those prompts to influence execution without a reliable trust separator. That is why the OWASP Agentic AI Top 10 and OWASP NHI Top 10 are increasingly relevant together. One addresses behaviour in agentic systems, the other addresses the credentials and identities those systems depend on. Practitioners should align prompt safety with identity containment.

LLM security debt accumulates when organisations scale usage before they can audit it. The article's operational guidance is sound, but the market signal is stronger: monitoring, logging, and detection-as-code are becoming the minimum viable control set because static policy cannot keep pace with model behaviour. LLM governance debt: the gap between how quickly teams deploy model-enabled workflows and how slowly they establish auditability, control boundaries, and response ownership. Practitioners should close that gap before expanding use cases.

Secrets exposure remains the shortest path from AI misuse to enterprise compromise. The most practical threat pattern in this category is not exotic model manipulation. It is compromised credentials attached to model infrastructure, retrieval stores, or agent tooling. That is why identity lifecycle, secret rotation, and offboarding matter directly to LLM security. When an AI system keeps access longer than its business purpose, the attack window widens. Practitioners should assume that every persistent secret attached to an AI workflow is a future incident candidate.

LLM governance must move from point controls to operating model design. Input filters and output classifiers help, but they do not solve the core problem if teams cannot answer who owns the model, who approves its tools, who reviews its access, and who responds when behaviour changes. That is a governance question, not just a security engineering one. Practitioners should build explicit accountability for model identity, delegated access, and incident response before scaling deployment.

What this signals

LLM programmes are moving from experimentation to delegated execution, which means the next control failure will more often look like over-permissioned identity than model failure. The immediate priority is to bring model access, retrieval scope, and secret handling into the same governance model used for privileged workloads, then align it with NIST AI Risk Management Framework and OWASP Agentic AI Top 10.

LLM governance debt: if teams cannot audit what a model can see, what it can call, and what it did last week, they do not have an AI operating model, only an exposure surface. The practical signal to watch is whether model ownership, prompt logging, and tool approval are becoming standard control points rather than ad hoc engineering choices.

As more organisations wire LLMs into production systems, identity lifecycle controls become more important than one-off prompt rules. The next maturity step is to treat secrets rotation, offboarding, and policy review as part of AI deployment, not as separate hygiene tasks.

For practitioners

• Define model identity and ownership Assign a named owner for each production LLM workflow, including the service account, retrieval layer, and downstream tools it can use. Make ownership part of the access review and incident response process so security teams know who can approve changes and who is accountable when behaviour drifts.
• Constrain delegated privileges Reduce each model's tool scope to the minimum required, prefer read-only access where possible, and use short-lived credentials for any action that can alter data or trigger workflows. Review the backing service account permissions as part of the same control, not separately.
• Instrument prompt and retrieval audit trails Log full prompt-response pairs, retrieved documents, tool calls, and policy decisions so investigators can reconstruct how an output was produced. Use these records to detect prompt injection patterns, abnormal retrieval paths, and repeated attempts to extract sensitive data.
• Treat outputs as untrusted content Validate every model response before it reaches a database, shell, ticketing system, or workflow engine. Apply the same scrutiny you would apply to untrusted user input, because downstream execution is where many LLM attacks become operational incidents.
• Rotate and offboard AI secrets quickly Inventory API keys, tokens, and certificates tied to model services, rotate them on a fixed schedule, and revoke them immediately when a workflow changes owner or purpose. Persistent secrets are a direct route from model misuse to broader compromise.

Key takeaways

• LLM security is an identity and privilege problem as much as a model safety problem, because tool access and delegated credentials turn models into attack paths.
• The evidence is already broad: prompt injection, credential abuse, data leakage, and model hijacking all show that traditional controls miss the way LLMs actually operate.
• Teams need governance that covers ownership, least privilege, logging, retrieval validation, and secret lifecycle management before scaling AI workflows further.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Retrieval-augmented Generation: Retrieval-augmented generation is a pattern where an AI model pulls external information before generating output. The security challenge is that access rules can weaken when data is chunked, embedded, cached, or reused, so source permissions may not automatically follow the content into the model's context.
• Model Tool Access: Model tool access is the set of APIs, workflows, and external actions an LLM can invoke while running. It is an identity and privilege issue because each tool expands the model's effective authority, making least privilege, approval boundaries, and logging essential controls.
• LLM Governance Debt: LLM governance debt is the gap between rapid deployment of model-enabled workflows and the slower work of defining ownership, access control, auditability, and response procedures. It accumulates when teams scale use cases before establishing the controls needed to manage model behaviour safely.

What's in the full article

Panther's full article covers the operational detail this post intentionally leaves for the source:

• Detection-as-code examples for LLM-specific threat patterns such as prompt injection and abnormal retrieval behaviour
• Practical control steps for filtering prompts, validating outputs, and monitoring model activity at scale
• Examples of how teams can instrument logs for prompt-response pairs, tool calls, and policy decisions
• Implementation advice for teams using model-enabled applications, assistants, or retrieval-augmented generation systems

👉 Panther's full article covers the risk categories, defensive patterns, and detection guidance in more operational detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and workload identity for practitioners building stronger access controls. It helps security teams connect identity discipline to the broader security programmes now shaping AI and application risk.