LLM security testing breaks the web-app playbook for MCP apps

At a glance

What this is: Escape shows that LLM applications need a different security testing method because model inputs, tool calls, and output sinks collapse into a single attack surface.

Why it matters: That matters to IAM and NHI practitioners because agent permissions, tool reach, and output handling now govern whether a model can disclose data, invoke privileged actions, or turn untrusted text into execution.

By the numbers:

• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, sharing sensitive data, and revealing access credentials.
• 17 minutes.

👉 Read Escape's full analysis of LLM security testing and MCP risk

Context

LLM security is not a minor extension of web application testing. Once a model can read natural language, call tools, and send output into a browser or database, the traditional separation between code and data no longer holds, and familiar controls can miss the real failure path. In AI security terms, the problem is governance of model inputs, tool reach, and output sinks.

This article is about that security gap, not a single vendor implementation. For IAM and NHI teams, the most relevant issue is that model agency is now tied to privileged access, tool permissions, and downstream data handling. That makes AI application testing part of identity governance, not just application security.

The starting position is increasingly typical in AI-enabled products: teams scope the model and miss the surrounding system. That is exactly where prompt injection, excessive agency, and output handling failures emerge, especially when MCP or similar tool interfaces are involved.

Key questions

Q: What breaks when prompt injection defenses are not in place for LLMs?

A: When prompt injection defenses are weak, attacker-controlled text can override the model's intended instructions, especially when the text arrives through retrieved content or external documents. The result is not always obvious compromise. It can be silent instruction hijacking, unsafe output, leaked context, or model behaviour that downstream systems trust too much.

Q: When does AI governance become an IAM and NHI problem?

A: It becomes an IAM and NHI problem as soon as autonomous systems use credentials, APIs, or delegated access to perform actions. At that point, the quality of identity assignment, privilege scope, logging, and lifecycle control determines whether the system can be governed and audited responsibly.

Q: How do security teams reduce the impact of unsafe LLM output handling?

A: They should block direct execution of model output until it has been validated, normalised, and checked against the intended action. This matters most when output can become SQL, shell commands, or code for downstream systems. The control is to treat the model as an untrusted source, not as an execution authority.

Q: How should security teams test MCP-connected AI systems for real risk?

A: Start with the trust chain, not the model output. Map every context source, then test whether authentication, encryption, validation, and tool authorization hold when inputs are malicious or altered. The goal is to prove that the agent cannot turn untrusted context into action, data exposure, or tool misuse.

Technical breakdown

Why LLM security testing is different from web application testing

Traditional application testing assumes a parser boundary between input and execution. LLM systems erase that boundary because the same natural-language stream can carry user instructions, embedded malicious text, system directives, and tool output. That makes evaluation probabilistic rather than binary. A single payload may succeed only some of the time, but that is still a live vulnerability if the model can be induced to call tools, expose prompts, or render unsafe output. Security testing therefore has to examine the full application around the model, not just the model prompt.

Practical implication: test agent workflows, not isolated prompts, and treat repeatable low-frequency hits as exploitable findings.

MCP server security and the trust boundary around tools

The Model Context Protocol moves the trust boundary into the text the model reads before it decides on tool use. Tool descriptions and tool responses both enter the model as trusted context, which means an attacker can poison either source and steer tool selection. The risk is not only prompt injection. It also includes confused deputy behaviour, token passthrough, session hijacking, and local server compromise. In practice, the model can become a policy bypass layer if untrusted text is allowed to influence privileged tool invocation.

Practical implication: review every MCP tool, response path, and local proxy as if it were an identity boundary.

Improper output handling turns model output into execution

One of the most dangerous LLM failure modes is output that lands in a browser, shell, or database without sanitisation. In the article’s example, the model emits Markdown and the frontend renders raw HTML, so malicious content executes when the next user opens the transcript. That is different from ordinary XSS because the payload is generated at runtime by the model, not submitted directly by a user. The control failure is at the sink, not the input filter. Security teams need to validate every render path and every downstream consumer of model output.

Practical implication: sanitise model output at every sink and add regression tests for browser-rendered transcripts.

Threat narrative

Attacker objective: The attacker wants to convert an apparently ordinary AI interaction into unauthorized data disclosure, tool misuse, or browser-side code execution.

1. Entry occurs when hostile natural language, tool descriptions, or poisoned content reaches the model through a normal conversation or retrieval path.
2. Escalation happens when the model treats that untrusted text as instruction and invokes a tool, reveals hidden context, or produces unsafe output.
3. Impact follows when output handling, privileged tool access, or browser rendering turns the model’s action into data disclosure, script execution, or downstream compromise.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM security is now an identity and access problem, not just an application-testing problem. Once a model can call tools, read files, and render output, its permissions become part of the trust model. That puts tool scope, session reach, and output sinks squarely into the governance layer, especially where NHI-style service accounts or API-backed agents sit behind the model. Practitioners should treat model-connected privileges as governed access, not as auxiliary application logic.

Indirect prompt injection creates a governance gap that conventional input filters cannot close. The article’s core lesson is that malicious instructions can arrive through documents, notes, web pages, or tool responses after initial validation has already happened. That means the control gap is not just sanitation, but runtime trust in content the model reads later. For identity teams, this is the same structural problem seen when untrusted delegates inherit more access than their lifecycle can safely contain.

Output handling is the named failure mode teams must start tracking. A model that emits Markdown into a raw HTML sink converts a language task into a browser execution risk. That is a specific, citable concept: output-sink exposure. Security programmes should measure whether generated content can reach executable contexts without sanitisation. The practical conclusion is simple: if the sink is unsafe, the model is unsafe.

MCP widens the attack surface by making tool trust visible to the model itself. The model does not just consume user text. It also consumes tool metadata and tool output, which means the control plane is now partly linguistic. This is where AI security and NHI governance intersect most sharply. Teams need access scoping, response filtering, and session isolation for every tool the model can reach, or the model becomes a privileged relay for untrusted instructions.

Regression testing is the only durable control when model behaviour changes underneath you. The article’s repeatable loop matters because a fixed exploit can disappear and reappear as the model, prompt, or toolchain changes. That makes one-off red teaming insufficient. Practitioners should freeze proven findings as tests and rerun them on every build. In a rapidly changing AI stack, governance without regression is only temporary assurance.

From our research:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, sharing sensitive data, and revealing access credentials, according to AI Agents: The New Attack Surface report.
• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.
• Forward look: Read OWASP Agentic AI Top 10 for the control patterns that help close this governance gap.

What this signals

Output-sink exposure will become a standard control question in AI application reviews. Teams will increasingly need to prove that model-generated content cannot execute in browsers, shells, or workflow engines without sanitisation and policy checks. That is the point where application security and identity governance meet, because the model’s authority now depends on the privileges and sinks it can reach.

AI programmes should expect indirect prompt injection to remain a persistent design issue rather than a one-time bug class. The practical response is to pair runtime monitoring with build-time regression tests, then tie tool permissions to explicit service identities and documented approval paths. That approach aligns best with NIST AI Risk Management Framework and the trust-boundary concerns highlighted in OWASP Agentic AI Top 10.

For practitioners

• Map every model sink and tool boundary Inventory where model output can land, including browser renderers, databases, shells, and ticketing systems, then classify which sinks require sanitisation before display or execution.
• Restrict MCP tool scope and response trust Review each MCP tool for least privilege, narrow the data it can return, and treat tool descriptions and tool responses as untrusted until they are validated by policy.
• Add regression tests for indirect prompt injection Convert every proven jailbreak, tool-poisoning path, or output-handling flaw into a build-time test that reruns on each model or prompt change.
• Sanitise generated content before browser rendering Strip raw HTML, block executable markup, and validate transcript renderers so model-generated Markdown cannot become active code in the browser.
• Tie model permissions to identity governance Give AI systems explicit service identities, review their access like any other privileged account, and revoke broad tool access that the model does not need for the task.

Key takeaways

• LLM applications fail when teams assume web-app controls still define the trust boundary, because model inputs, tool calls, and outputs now share one attack surface.
• The article shows that prompt injection, unsafe output handling, and excessive agency can all become exploitable when the surrounding system is not governed as part of the model.
• Practitioners should respond with least-privilege tool scope, sanitised sinks, and regression tests that keep proven LLM exploits from reappearing.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Mcp Trust Boundary: The security boundary created by a Model Context Protocol connection between an AI assistant and an external system. Each connection can supply data, trigger actions, or both, so it must be governed like a delegated access path rather than a simple integration.
• Output-Sink Exposure: Output-sink exposure occurs when generated model content reaches a browser, database, shell, or workflow engine without adequate sanitisation or policy checks. The risk is not the text alone, but the execution context it lands in, which can convert model output into script execution or data corruption.
• Excessive agency: A condition where an AI system is given more operational authority than its task requires. The risk is not just poor output. It is that mistakes, manipulation, or compromise can produce destructive actions at machine speed across the systems the agent can reach.

What's in the full article

Escape's full analysis covers the operational detail this post intentionally leaves for the source:

• A repeatable LLM pentesting loop that maps recon, exploitation, proof, and regression into a testable workflow.
• A vulnerable MCP lab with code-level examples for tool poisoning, indirect prompt injection, and unsafe file access.
• Practical guidance on measuring probabilistic findings as hit rates instead of pass or fail.
• Source-level detail on how browser sinks, tool outputs, and prompt updates change the risk profile after deployment.

👉 Escape's full article covers the lab setup, attack chain, and regression-test workflow in more operational detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, workload identity, and agentic AI identity. It helps security and identity practitioners connect access control, lifecycle management, and governance across modern AI-enabled environments.