LLM app security risks show where GenAI controls fail

At a glance

What this is: This is an analysis of LLM application security risks, highlighting that GenAI failures cluster around prompt control, credential exposure, tool permissions, and retrieval boundaries.

Why it matters: It matters because IAM, PAM, and NHI teams increasingly have to govern LLMs as privileged actors that can leak secrets, overreach permissions, or amplify bad data.

Context

LLM application security fails when teams treat the model as the main risk object and ignore the control boundary around it. The real security problem is not whether the model sounds correct, but whether prompts, retrieved context, tools, and framework settings can be manipulated into doing work the system was never meant to permit. In practice, that makes LLM security an IAM and governance problem as much as an application security problem.

Flatt Security’s analysis maps those failure modes to the OWASP Top 10 for LLM Applications 2025 and shows how the risk shifts when an LLM moves from chat-only use into RAG, tool use, or framework-driven orchestration. Once the model can read data, call services, or act on behalf of a user, the boundary between human identity, application identity, and machine identity becomes operationally important.

The starting position in this article is typical of modern GenAI deployments: the security issues emerge first in development choices, then become governance gaps once the application is connected to data or action-oriented tools.

Key questions

Q: What breaks when LLM applications do not enforce a clear failure boundary around prompts and tools?

A: The application breaks at the delegation layer. Malicious prompts, poisoned retrieval content, or unsafe templates can push the model into actions the developer never intended, especially when tools, files, or external APIs are available. The result is not just a bad answer but a security failure that can expose data, alter records, or trigger unsafe execution.

Q: Why do overprivileged LLM tools create more risk than prompt filtering alone can address?

A: Because the model’s permissions determine the damage it can do after the prompt is accepted. Prompt filtering may reduce abuse, but if the model can write files, call APIs, or access sensitive data, one successful injection can become a high-impact incident. The risk is delegated authority, so the control point is the permission boundary, not only the input filter.

Q: What are the signs that an LLM is failing basic governance controls?

A: Warning signs include inconsistent responses to similar prompts, weak refusal behavior, uncontrolled exposure of sensitive inputs, and poor visibility into post-deployment activity. A model can also be considered poorly governed when teams cannot explain where it is used, what it can access, or how outputs are reviewed. Governance fails first in the gaps between deployment, monitoring, and access control.

Q: Should organisations treat RAG controls and prompt controls as the same problem?

A: No. Prompt controls manage how instructions influence the model, while RAG controls manage what data is allowed into the model’s context. If teams collapse them into one control area, they miss provenance checks, access restrictions, and poisoning risks in retrieval pipelines. Good governance separates instruction safety from context safety.

Technical breakdown

Prompt injection turns natural language into an attack surface

Prompt injection occurs when attacker-controlled text changes how the model interprets instructions, causing it to ignore intended constraints or reveal information. That can be direct, where the user supplies the malicious instruction, or indirect, where the model ingests hostile content from a webpage, document, or search result. The core issue is that LLMs do not reliably separate instructions from data. Once an application lets the model act on the resulting output, the injection becomes a control bypass rather than a simple content problem.

Practical implication: constrain what the model is allowed to do after prompt processing, not just what it is allowed to say.

System prompts and tool permissions create hidden identity risk

System prompts often hold role instructions, safety rules, and sometimes sensitive configuration material. If secrets, internal logic, or access details are embedded there, leakage can expose both operational knowledge and security boundaries. The risk becomes sharper when the LLM also has tool access, because the model may inherit permissions that belong to the surrounding application or user session. In that sense, the model behaves like a non-human actor whose authority must be scoped explicitly, not assumed from the interface.

Practical implication: keep secrets out of prompts and assign the LLM only the minimum tool permissions required for the task.

RAG and framework configuration fail when trust is not enforced at ingestion time

Retrieval-augmented generation expands the attack surface by letting external or internal content shape the model’s answers. If vector stores, embeddings, or source corpora include untrusted or overly broad data, the model can surface other users’ information, poisoned content, or instructions that alter downstream behaviour. Frameworks add another failure mode when unsafe template engines or permissive file or function settings allow code execution or uncontrolled file access. The weakness is not the model alone, but the surrounding pipeline that decides what context and capability it receives.

Practical implication: validate trust and access rules before data reaches the retrieval layer or the framework runtime.

Threat narrative

Attacker objective: The attacker wants to make the LLM disclose sensitive context, perform unauthorized actions, or execute unsafe code through the surrounding application.

1. Entry occurs when a malicious prompt, poisoned document, or unsafe template reaches the LLM through chat input, retrieved content, or framework configuration.
2. Credential or authority abuse follows when the model is allowed to see secrets, internal rules, or tool permissions that should not be exposed to the requestor.
3. Escalation happens when the LLM uses overbroad access to write, delete, retrieve, or generate content beyond the original user intent.
4. Impact is data leakage, unintended actions, denial of service, or code execution through the application stack.

Breaches seen in the wild

• EchoLeak (Microsoft 365 Copilot) 2025: A crafted email could make Microsoft 365 Copilot leak data from its context with no click, a zero-click prompt injection fixed as CVE-2025-32711.
• Meta Muse agent hijack 2026: An undocumented Muse setting let local malware hijack Meta's personal AI agent, steal its authentication material and abuse user access.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM security is really governance over delegated authority: the article shows that the decisive question is not whether the model is accurate, but whether it is allowed to do anything at all. Once prompts, tools, and retrieval layers can trigger external action, the model behaves like a non-human identity with runtime authority. Practitioners should treat that authority as a governed access surface, not as an emergent property of the model.

System prompt leakage exposes a control-plane mistake, not just a data leak: if instructions, rules, or secrets are stored in prompts, the application has placed security logic inside an object that the attacker can influence. That failure mode is structurally similar to hardcoding trust decisions into user-controllable input. The right lesson is that policy must live outside the prompt boundary, because the prompt boundary is not a safe enforcement point.

RAG expands the attack surface by turning data quality into security quality: when retrieval feeds model behaviour, poisoned content and overly broad context access become governance issues, not merely retrieval bugs. This creates a named failure mode we can call context trust collapse: the application can no longer distinguish sanctioned knowledge from attacker-influenced content. Practitioners should assume retrieval must be access controlled, provenance aware, and continuously monitored.

Excessive agency is the LLM equivalent of overprivileged service access: broad write, delete, or external system permissions turn a language interface into a high-impact actor. That is why this topic belongs beside PAM and NHI governance, especially where the model can call APIs or alter records. Teams should limit agency at the point of delegation, because post-hoc review cannot reliably undo an unsafe action already taken by the system.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Generative AI use specifically increased from 33% in 2023 to 79% in 2025, according to McKinsey’s Global Surveys on the State of AI.
• Read next: Agentic AI Security Guide

What this signals

Context trust collapse: LLM programmes fail when teams assume retrieved data is inherently safe because it came from an internal system. Security teams should separate prompt safety, retrieval provenance, and tool authorisation, because those are distinct trust decisions with different failure modes.

A strong GenAI control model treats the LLM as a delegated actor whose permissions are smaller than the surrounding application and whose context is continuously filtered. That is the right mindset for IAM, PAM, and NHI governance when models can read, retrieve, or act on behalf of users.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026. That pattern reinforces the need to govern credentials, not just prompts, across the full AI stack.

For practitioners

• Constrain model actions before deployment Define the exact operations the LLM may perform, then block any tool, API, or workflow step that is outside the approved task scope.
• Remove secrets from prompts and context Keep API keys, tokens, and internal rules outside system prompts, retrieval payloads, and template variables so a prompt leak cannot expose them.
• Apply least privilege to LLM-connected tools Grant read-only access where possible, separate write actions into distinct controls, and review any workflow that lets the model change records or files.
• Harden retrieval and template inputs Treat RAG sources, embeddings, file inputs, and template syntax as untrusted, and validate provenance before content reaches the model runtime.

Key takeaways

• LLM application risk is driven by delegated authority, not model output quality alone, because prompts, retrieval, and tools can all become attack paths.
• The article shows that system prompts, RAG context, and framework settings can expose secrets, widen permissions, or enable unsafe execution.
• Teams need separate controls for prompt safety, context provenance, and tool authorisation if they want GenAI governance to hold up in production.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• System Prompt Leakage: System prompt leakage is the exposure of hidden prompt content to users or attackers. The real security problem is usually not the prompt itself, but the secrets, policy logic, and internal architecture details placed inside it. If those details are sensitive, they should live in code or secrets management instead.
• Retrieval-augmented Generation: Retrieval-augmented generation is a pattern where an AI model pulls external information before generating output. The security challenge is that access rules can weaken when data is chunked, embedded, cached, or reused, so source permissions may not automatically follow the content into the model's context.
• Excessive agency: A condition where an AI system is given more operational authority than its task requires. The risk is not just poor output. It is that mistakes, manipulation, or compromise can produce destructive actions at machine speed across the systems the agent can reach.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps security practitioners build the control boundaries that GenAI systems need without confusing model behaviour with identity governance.