LLM security testing breaks the web-app playbook for MCP apps

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Escape's blog post: LLM apps need a different security testing model because prompt injection, tool access, and output rendering collapse the usual web-app boundary between code and data.

Why it matters: IAM and security teams need to treat AI outputs, tool permissions, and downstream sinks as governance surfaces, because one compromised instruction path can create execution or disclosure risk across the wider system.

Context

LLM application security is not just another web-app testing problem. Once a model can read instructions, call tools, and place output into browsers or databases, the security boundary shifts from input validation to system governance, especially where MCP expands what the model can reach.

That matters for identity and access management because the model is effectively an actor with delegated reach. If tool descriptions, tool responses, and rendered outputs are all trusted text, then privilege, data access, and output handling must be governed as one chain rather than separate controls.

Key questions

Q: What breaks when application security testing does not cover LLM-specific threats?

A: Testing gaps allow issues unique to AI workloads to reach production, including prompt injection, data leakage, unsafe tool use, and improper access to downstream systems. Traditional API tests may confirm basic authentication and input validation, yet still miss model behaviour, context abuse, and indirect data exposure. The result is incomplete assurance and a false sense of control.

Q: Why do excessive tool permissions make prompt injection more dangerous in MCP apps?

A: Prompt injection becomes materially worse when a model can call tools that touch files, APIs, or browsers. The injection no longer just changes a reply, it can steer a delegated actor into reading data, moving laterally, or writing active content to a sink. The risk is privilege, not just text manipulation.

Q: How do security teams know whether an LLM is operating safely?

A: Look for evidence that the model’s read scope, tool scope, and output handling are all bounded and reviewed. If the system can only act on approved data, only invoke approved tools, and logs those actions clearly, governance is working. If any of those are missing, the model is outside control.

Q: Should organisations treat proven LLM exploits as regression tests?

A: Yes. LLM systems are non-deterministic and change as prompts, tools, and model versions change, so a one-time finding can reappear later. Capturing the exact exploit path as a regression test turns a security observation into an ongoing control and stops silent reintroduction after updates.

Technical breakdown

Why prompt injection defeats parser-based assumptions

Traditional web security relies on a parser separating code from data. LLM applications do not preserve that boundary because natural language instructions, retrieved content, and system prompts are all fed into the same model context. That makes prompt injection possible even when the input looks harmless, since the model cannot reliably distinguish user intent from embedded instructions in documents or tool output. The problem becomes stronger in indirect prompt injection, where the attacker plants text in content the model reads later, not in the original user request. Practical implication: test the full instruction flow, not just user input fields.

Practical implication: validate every place untrusted text enters model context, including retrieved documents and tool responses.

How MCP widens the trusted text boundary

MCP gives the model real tool access, so the trust boundary moves into the protocol layer. Tool descriptions and tool responses are both read by the model as trusted context, which means they can carry hidden instructions as easily as they carry data. If a tool can also reach files, APIs, or browser-like actions, then a successful injection can become data disclosure or operational abuse without a conventional auth bug in the application code. The architectural issue is not the presence of a tool, but the fact that the model can be steered by text attached to that tool. Practical implication: review tool metadata, tool output, and privilege scope as one security object.

Practical implication: treat MCP tool metadata, tool output, and permission scope as a single governed trust boundary.

Why output sinks turn model mistakes into browser exploits

Output handling is a distinct attack surface in LLM systems because the model can generate active content at runtime. In the article's examples, Markdown emitted by the model was rendered with raw HTML enabled, so the browser executed stored XSS when someone later viewed the transcript. This is different from classic reflected input attacks because the payload may be created on demand by the model after testing has started. The sink, not the prompt, becomes the decisive control point. Practical implication: sanitise and constrain every downstream sink that renders, executes, or stores model output.

Practical implication: enforce output sanitisation and safe rendering before model text reaches browsers, shells, or databases.

Threat narrative

Attacker objective: The attacker wants to steer the model into revealing sensitive context, invoking privileged tools, or producing output that executes in the next viewer's browser.

1. Entry occurs when hostile text is placed into content the model will later read, such as a document, note, issue, or tool description.
2. Credential or data access follows when the model is induced to call a tool, read an out-of-scope file, or disclose its own prompt and session context.
3. Impact lands when the model's generated output is rendered into a browser or passed into a sensitive sink, turning trusted text into disclosure or execution.

Breaches seen in the wild

• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.
• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM security testing is a trust-boundary discipline, not a prompt-filtering exercise. The article shows that the real control points sit around the model, not inside it. Instructions, tool metadata, retrieved content, and rendered output all behave like one governance surface, so the testing model has to follow the full decision chain. The practitioner conclusion is that security assurance for LLM applications starts with trust-boundary mapping.

MCP creates a delegated identity problem as much as a model-risk problem. Once the model can call tools, it is operating like a non-human actor with borrowed reach. That means tool scope, response handling, and session context need the same discipline identity teams apply to privileged service accounts and other NHIs. The practitioner conclusion is that AI tool access should be governed as delegated privilege, not convenience plumbing.

Output handling is the overlooked control that turns AI misuse into real compromise. The stored XSS example shows that model-generated text can become active content when it reaches the wrong sink. That makes sanitisation, rendering policy, and sink-specific controls part of the security model, not just frontend hygiene. The practitioner conclusion is that every model output path needs explicit control ownership.

Regression-proofing proven exploits should become a standard AI security pattern. The article's repeatable loop matters because LLM behaviour changes under model updates, prompt changes, and tool changes. A finding that cannot be replayed after the next release was never governed, only observed. The practitioner conclusion is to turn each validated exploit into a permanent test case tied to the system version and sink.

From our research library:

• 24,008 unique secrets were exposed in MCP configuration files in 2025 alone, the protocol's first year of widespread adoption, according to the State of Secrets Sprawl 2026.
• Read next: MCP Security Guide

What this signals

Prompt injection becomes a governance problem once tool access and output sinks sit on the same chain. Security teams should expect the weak point to move from the prompt itself to the surrounding system design, especially where MCP expands what the model can touch. The practical response is to govern model reach, not just model text.

Stored AI output needs the same scrutiny as any other untrusted content. If a chat transcript, assistant reply, or generated note can reach a browser with raw HTML enabled, the model has effectively become a content injector. Teams should treat output sanitisation, render policy, and sink ownership as core AI controls.

MCP configuration files are now part of the secret-exposure problem set. 24,008 unique secrets were exposed in MCP configuration files in 2025 alone, according to the State of Secrets Sprawl 2026. That scale means discovery, inventory, and least-privilege reviews for MCP servers are no longer optional programme hygiene.

For practitioners

• Map every model trust boundary Inventory where untrusted text enters the model, where tool responses return, and where output is rendered or stored. Include MCP tool descriptions, retrieved documents, and downstream sinks in the same review.
• Restrict tool scope to the minimum necessary Limit which tools the model can call, which files it can read, and which network or browser actions it can trigger. Separate high-risk tools from ordinary chat flows so excessive agency cannot reach sensitive systems by default.
• Sanitise model output before any render path Treat Markdown, HTML, and other model-generated markup as untrusted until it has passed sink-specific sanitisation. Apply stricter handling to transcript viewers, admin consoles, and any UI that can execute raw HTML.
• Convert proven exploits into regression tests Record the exact prompt, tool path, and sink that produced the leak, then run that case on every build and model change. Keep the exploit alive as a test so a later update cannot silently reopen it.
• Test indirect injection paths explicitly Plant benign-looking content in notes, documents, issues, or retrieved pages and verify whether the model follows hidden instructions from that content. Measure hit rate, not just one-off success, because LLM behaviour is probabilistic.

Key takeaways

• LLM applications break the web-app testing model because the model, its tools, and its outputs share a single trust boundary.
• Prompt injection becomes materially more dangerous when the model can reach tools, files, or browser-rendered sinks with delegated privilege.
• Security teams should turn each proven exploit into a regression test so model changes do not silently reopen the same path.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Mcp Trust Boundary: The security boundary created by a Model Context Protocol connection between an AI assistant and an external system. Each connection can supply data, trigger actions, or both, so it must be governed like a delegated access path rather than a simple integration.
• Excessive agency: A condition where an AI system is given more operational authority than its task requires. The risk is not just poor output. It is that mistakes, manipulation, or compromise can produce destructive actions at machine speed across the systems the agent can reach.
• Improper Output Handling: A vulnerability where LLM-generated content is trusted before it reaches another system. The danger appears when that output is rendered, parsed, or executed without validation, encoding, or policy checks, allowing attackers to turn model responses into code, queries, or commands.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management for practitioners building durable identity controls. It helps security teams connect delegated access, lifecycle control, and operational assurance across modern identity programmes.