OWASP LLM vulnerabilities expose gaps in enterprise AI governance

At a glance

What this is: Kong’s guide maps the OWASP Top 10 for LLM applications to the enterprise AI security problems that matter most, especially prompt injection, data leakage, and RAG abuse.

Why it matters: IAM, NHI, and AI governance teams need to treat LLMs as a distinct control plane because identity, authorization, and observability all break down when AI systems can be manipulated through language.

By the numbers:

• 72% anticipate increased LLM spending in the coming year.
• About 40% are already investing more than $250,000 USD per year.
• OWASP’s Top 10 for LLM Applications 2025 was released in November 2024.

👉 Read Kong's analysis of OWASP Top 10 LLM vulnerabilities and AI gateway controls

Context

LLM security is no longer a side topic of application risk. Once natural language becomes a control surface, prompt text, retrieval sources, model outputs, and downstream actions all become part of the security boundary, which means traditional web app controls only cover part of the problem.

That shift matters for identity programmes because LLM systems often sit between users, services, and data sources. The governance challenge is not just whether access is granted, but whether the model can be induced to use that access in unintended ways through prompt manipulation, retrieval poisoning, or unbounded tool consumption. For teams standardising AI controls, the OWASP Agentic AI Top 10 is a useful companion reference, especially where LLMs feed agentic workflows.

Key questions

Q: How should security teams control prompt injection in enterprise LLMs?

A: Security teams should control prompt injection by treating prompts as untrusted input, filtering malicious patterns before model execution, templating system instructions separately from user content, and constraining what external data the model can ingest. The most effective programmes place these controls at the gateway or orchestration layer so they apply consistently across applications and models.

Q: Why does retrieval-augmented generation create new governance risk?

A: Retrieval-augmented generation creates new governance risk because the model can only be as trustworthy as the knowledge sources it retrieves from. If documents, embeddings, or vector stores are poisoned or too broadly exposed, the model may generate confident but wrong answers from unverified context. That makes source governance and provenance tracking essential.

Q: What do organisations get wrong about AI security monitoring?

A: Organisations often monitor only model output and miss the upstream signals that show abuse, such as repeated retries, unusual token consumption, or changes to retrieval sources. Effective AI monitoring has to cover prompts, data access, tool calls, and usage patterns together, otherwise security teams see the symptom after the control failure has already happened.

Q: How can teams decide whether to prioritise AI guardrails or traditional app controls?

A: Teams should prioritise AI guardrails when the model can influence decisions through natural language, external retrieval, or tool use, because traditional app controls do not reliably separate safe input from malicious instruction in that environment. Standard application security still matters, but it is insufficient on its own once the model becomes part of the decision path.

Technical breakdown

Prompt injection turns natural language into an attack path

Prompt injection works because LLMs treat language as instruction input, not as a rigidly typed command set. Attackers can embed malicious directives directly in a user prompt or indirectly in retrieved content, causing the model to ignore system instructions, reveal sensitive context, or take harmful actions. The core failure is that security controls designed for deterministic software do not reliably separate trusted instructions from untrusted language when both look similar at runtime. In enterprise settings, the risk increases when the model is allowed to process external documents, emails, tickets, or web content.

Practical implication: place content filtering, prompt shaping, and output controls at the gateway layer, not only inside application code.

RAG and vector database exposure widen the AI attack surface

Retrieval-augmented generation introduces a second trust boundary: the knowledge base. If embeddings are poisoned, documents are tampered with, or vector stores are overexposed, the model can retrieve and reuse malicious context as if it were legitimate evidence. That changes the attack from simple prompt tampering to knowledge manipulation. The security problem is not only whether the model answers correctly, but whether the retrieval layer can be influenced to supply the wrong grounding material. Once that happens, the model may appear confident while being operationally wrong.

Practical implication: apply access controls, provenance checks, and monitoring to the retrieval layer with the same seriousness as the model endpoint.

Uncontrolled resource use is a governance and cost problem

LLM systems can be driven into excessive token consumption, repeated retries, or repeated external calls, which turns a security issue into a resilience and cost issue. Abuse may be intentional, as in prompt flooding or automated probing, or accidental, as in poorly bounded workflows. Because AI requests often cascade across models, tools, and plugins, a single user interaction can create a much larger runtime footprint than a standard API call. That makes usage limits and observability part of the security architecture, not just the finance stack.

Practical implication: set rate limits, budgets, and telemetry thresholds for AI traffic before broad production rollout.

Threat narrative

Attacker objective: The attacker wants the model to reveal information, follow harmful instructions, or distort downstream decisions while appearing to operate normally.

1. Entry occurs when a malicious prompt, embedded document, or external content reaches the LLM through a user-facing or machine-facing interface.
2. Escalation occurs when the model follows the injected instruction, retrieves restricted context, or expands its response beyond the intended task boundary.
3. Impact occurs when the system leaks sensitive data, makes unsafe decisions, or consumes resources in ways that bypass normal application expectations.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM security is now an identity governance problem, not just an application security problem. Once a model can consume prompts, context, and external data, the governance question becomes who or what is authorised to influence that decision path. That is why prompt injection, retrieval poisoning, and uncontrolled resource use should be read as access-control failures in disguise. Practitioner conclusion: model security programmes need identity boundaries around inputs as well as outputs.

Prompt-based manipulation exposes a control gap that traditional deterministic security never had to solve. Legacy app controls assume commands and content are separable, but LLMs collapse that distinction by design. The result is that the trust boundary moves from syntax to semantics, which is harder to enforce and harder to audit. Practitioner conclusion: teams should treat prompt handling as a governed security interface, not a developer convenience.

Vector databases create an identity and provenance problem at the retrieval layer. If a model can retrieve from unverified or overbroad sources, the system is effectively delegating authority to content it has not authenticated. That makes retrieval governance as important as model selection. Practitioner conclusion: organisations should define which knowledge sources are trusted, who can modify them, and how those changes are audited.

Uncontrolled AI resource consumption is an operational signal that governance is incomplete. When prompts can trigger repeated inference, retries, or tool calls, the model can create cost, latency, and availability incidents without a classic breach. That broadens the control objective from confidentiality to resilience. Practitioner conclusion: AI programme owners should manage usage ceilings and observability as part of security governance, not as afterthoughts.

OWASP’s LLM framework is becoming the common language for enterprise AI risk. By naming risks such as prompt injection, sensitive information disclosure, and vector weaknesses, it gives security teams a way to align engineering, risk, and identity stakeholders around the same failure modes. Practitioner conclusion: teams that adopt a shared taxonomy will mature faster than teams debating whether AI risk belongs to security, data, or platform ownership.

From our research:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%), according to AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
• For a broader view of how these risks are surfacing in practice, see OWASP Agentic AI Top 10 for the adjacent control model that many enterprises are now adopting.

What this signals

Prompt controls will increasingly be judged as policy controls, not just model safety features. As enterprise AI adoption expands, security teams will need to show who can submit prompts, which sources can be retrieved, and what outputs are allowed into downstream systems. That makes governance artefacts, audit trails, and routing policy more important than isolated model tuning.

AI programmes that separate model risk from identity risk will move too slowly. The practical boundary is already blurred because prompts, retrieval access, and tool invocation are all forms of authorisation. Teams that connect AI security to identity governance, especially around access review and data provenance, will have a better chance of controlling blast radius as deployments scale.

Review AI adoption against the OWASP Agentic AI Top 10 now, even if your current use cases are not fully agentic. Today’s assistant becomes tomorrow’s delegated workflow, and the control assumptions change quickly when the system starts selecting tools or acting on retrieved context. Early alignment reduces rework when AI moves from pilot to production.

For practitioners

• Map LLM trust boundaries end to end Document where user prompts, retrieved content, system prompts, tool calls, and model outputs cross security boundaries. Make each boundary explicit in architecture reviews so identity, data, and platform owners can assign control ownership.
• Enforce gateway-level prompt and content controls Apply filtering, templating, and policy checks before prompts reach the model and before model output reaches business workflows. This reduces reliance on application code paths that are harder to standardise across teams.
• Treat retrieval sources as governed assets Limit which vector databases, documents, and embedding sources can feed production models, then track changes to those sources with audit trails. Retrieval governance should be reviewed with the same discipline as privileged data access.
• Set usage ceilings for AI traffic Define rate limits, token budgets, and escalation thresholds for model calls, especially where tools or plugins can multiply request volume. Include these limits in resilience testing so abuse and runaway loops are visible before production expansion.

Key takeaways

• LLM vulnerabilities are fundamentally governance problems because prompt, retrieval, and output paths all shape who can influence model behaviour.
• Kong’s article shows that enterprise AI adoption is scaling faster than most security models, with 72% expecting higher spend and 40% already above $250,000 a year.
• Security teams should move controls to the AI boundary, where prompts, context sources, and usage limits can be governed consistently across applications.

Key terms

• Prompt Injection: Prompt injection is the manipulation of an LLM by embedding instructions that override or distort its intended behaviour. The model treats language as executable guidance, so the attack succeeds when untrusted content is allowed to influence the decision path without proper isolation or policy enforcement.
• Retrieval-Augmented Generation: Retrieval-augmented generation is a pattern where a model fetches external context before answering. It improves relevance, but it also introduces a governance boundary around the retrieval source, because poisoned or overly broad knowledge bases can shape the model’s output as strongly as the prompt itself.
• Vector Database: A vector database stores embeddings so models can retrieve semantically similar content quickly. In AI security, it is not just a search layer. It becomes a sensitive governance point because the documents, embeddings, and permissions behind it determine what context the model is allowed to trust.
• Runtime Observability: Runtime observability is the ability to see what an AI system is doing while it is operating, including prompts, tool calls, retrievals, and outputs. It is essential for AI governance because without it, security and compliance teams cannot reliably investigate misuse or prove control effectiveness.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or governance in your organisation, it is worth exploring.

This post draws on content published by Kong: Securing Enterprise AI: OWASP Top 10 LLM Vulnerabilities Guide. Read the original.