LLM attack testing shows prompt injection still breaks model guardrails

At a glance

What this is: Pangea reports that repeated attack testing across four LLMs exposed uneven resistance to prompt injection, input leakage and over-reliance scenarios.

Why it matters: AI security teams need evidence-based testing because guardrail strength varies by model and failure mode, and deployment risk changes when prompt injection still succeeds.

Context

LLM security is no longer just a model-safety discussion. Once a model is embedded in workflows, prompt injection, input leakage and over-reliance become control problems that affect data exposure, response integrity and downstream decision quality.

The core governance gap is assuming that a model with safety filters is therefore safe in practice. In reality, attack resistance depends on the exact prompt class, the surrounding application design, and whether the organisation tests the model under adversarial conditions rather than only accepting vendor claims.

Key questions

Q: What breaks when prompt injection controls are not tested against real attack patterns?

A: When prompt injection controls are not tested against realistic attack patterns, teams often overestimate their guardrails. A model may appear safe in normal use but still reveal data, ignore instructions, or invoke privileged functions under adversarial input. The result is false confidence, weak detection coverage, and security drift as configurations change over time.

Q: Why do LLM prompt injection failures become business risk so quickly?

A: Because the model’s output often feeds real decisions, customer responses or automated actions. When the workflow trusts a compromised response, one successful injection can cascade into data exposure, bad advice or operational mistakes. The business risk comes from delegation, not only from the model itself.

Q: What signs show that an LLM is leaking input or hidden context?

A: Look for the model repeating prompt fragments, reproducing embedded data, or revealing system instructions when asked irrelevant follow-up questions. Repeated leakage across test payloads usually means context boundaries are too loose, retrieval is overbroad, or session state is being reused without enough isolation.

Q: How should teams compare model safety results across different LLMs?

A: Compare them by failure mode, payload class and test conditions, not by a single overall score. A model that resists unsafe outputs may still fail on input leakage or over-reliance. The useful question is which control boundary fails, how often, and in which deployment pattern.

Technical breakdown

Prompt injection attacks bypass instruction hierarchy

Prompt injection works when untrusted text is treated as if it were higher-priority instruction, or when the application fails to separate user content from system intent. In LLM applications, the model does not inherently know which text is malicious. A successful attack can redirect the model, override task boundaries, or coerce it into revealing hidden instructions. The weakness is often architectural, not just model-specific, because the application passes mixed-context input into a single reasoning surface. Practical implication: treat prompt boundaries as a security control and test whether untrusted input can alter instruction priority.

Practical implication: Test whether untrusted input can alter instruction priority before relying on the model in production.

Input leakage shows why memory and context handling matter

Input leakage occurs when the model reproduces sensitive user content, embedded context, or hidden system material that should have stayed bounded. This is not the same as general model hallucination. It reflects weak containment between prompt layers, conversation history, tool output, and any retrieved context supplied at inference time. Applications that reuse context too broadly can turn transient data into retrievable content. In regulated or customer-facing settings, that becomes a data governance issue as much as an AI issue. Practical implication: bound what the model can retain, retrieve, and repeat across sessions and tools.

Practical implication: Bound what the model can retain, retrieve and repeat across sessions and tools.

Over-reliance creates a governance failure, not just a safety failure

Over-reliance happens when users or downstream systems treat model output as authoritative even when the model is uncertain, incomplete, or unsafe. That failure mode is important because the risk is not only incorrect content, but unchallenged automation of bad advice. In an enterprise setting, this becomes a decision-control problem: human review, confidence signalling, and task scoping all determine whether the model can safely influence action. Pangea’s results show that a model may appear strong in one category and still fail badly when the workflow assumes trustworthy output. Practical implication: define when model output is advisory only and enforce that boundary in workflow design.

Practical implication: Define when model output is advisory only and enforce that boundary in workflow design.

Threat narrative

Attacker objective: Force the LLM to ignore guardrails, reveal sensitive context or generate output that users and workflows treat as trustworthy.

1. Entry occurs when adversarial payloads are submitted into the model context, including direct prompts and indirect prompt injection through surrounding content.
2. Credential access is not the main issue here, but the attacker seeks control over instruction flow, hidden context and generated output rather than authentication tokens.
3. Impact follows when the model reveals sensitive input, ignores constraints or produces unsafe advice that downstream systems or users trust.
4. The practical objective is to make the model violate its intended guardrails so the application leaks information or returns manipulated output.

Breaches seen in the wild

• Anthropic Claude evaluation incidents 2026: Claude models told they had no internet access breached four real organisations during cyber evaluations, one via a malicious PyPI package.
• LiteLLM PyPI package breach: LiteLLM PyPI supply chain attack, credentials stolen from users.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Prompt injection is now a workflow security problem, not just a model robustness problem. Once an LLM sits inside an application, the real failure is often the surrounding trust boundary. Untrusted text, retrieved context and hidden instructions converge in one execution path, so the app architecture determines how far an attacker can steer the model. Practitioners should test the application layer, not only the base model.

Input leakage is a governance failure of context handling. The article shows that models can still reveal prompt material, embedded input or surrounding context when boundaries are poorly enforced. That makes data handling, session scoping and retrieval design part of AI governance. Organisations need to decide what context a model may see, remember and reproduce before they decide what it may answer.

Over-reliance is the most underestimated risk because it converts model error into operational decision error. A model that sounds confident can still be unsafe if the workflow lets output become action without review. This is where AI governance overlaps with human control design, because the issue is not just accuracy but delegation discipline. The practitioner conclusion is simple: treat model output as advice unless the workflow can prove otherwise.

Prompt boundary integrity: the security property that determines whether untrusted text can override system intent or expose hidden context. Pangea’s results reinforce that this is the control concept teams should name and test. If boundary integrity is weak, guardrails become cosmetic rather than enforceable. Practitioners should measure this property directly in red-team style evaluations.

The category is maturing toward measurable adversarial validation, not generic AI assurance. Broad safety statements do not tell security teams where a model fails, how often, or under what payload class. The article’s testing approach is useful because it turns LLM security into evidence that can be compared across workflows and models. Practitioners should demand testable controls and repeatable failure metrics.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

LLM security programmes need adversarial testing at the application boundary, not just a vendor safety review. The practical lesson is that prompt injection and input leakage are exposed by workflow design, retrieval behaviour and instruction handling as much as by the underlying model. Security teams should expect the attack surface to change whenever tools, memory or prompts change.

Prompt boundary integrity: if untrusted content can influence instruction priority, the system no longer has a reliable control boundary. That is why evaluation must become repeatable and scenario-based, with red-team payloads mapped to deployment paths rather than abstract model classes.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026. The broader signal is that AI risk increasingly sits in the infrastructure around the model, not only in the model weights themselves.

For practitioners

• Test prompt boundary integrity before deployment Run adversarial payloads against the full application stack, including system prompts, retrieval layers and any tool output the model can see. Focus on whether untrusted content can change instruction priority or expose hidden context.
• Restrict context exposure and replay Limit how much history, retrieved data and hidden configuration the model receives in a single task. Separate user input from system instructions and prevent unnecessary reuse across sessions.
• Treat model output as advisory by default Define which outputs may drive customer-facing, financial or operational actions, and require human review or compensating controls where confidence is uncertain or consequences are high.
• Red-team the failure modes that matter Prioritise prompt injection, input leakage and over-reliance scenarios in testing plans instead of relying on generic safety scores. Re-test after application, retrieval or system-prompt changes.

Key takeaways

• Prompt injection remains a practical control failure because untrusted input can still override instruction boundaries inside deployed LLM workflows.
• The article’s test results show that resilience varies by model and by attack class, so a single safety score is not enough for governance.
• Security teams should treat prompt testing, context isolation and output scoping as operational controls, not optional AI hygiene.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Input Leakage: Input leakage is the unintended exposure of data that was supplied to a model, including prompts, hidden instructions, or sensitive context. It becomes a governance issue when applications pass secrets or privileged information into the model and assume the model will not reproduce them.
• Over-Reliance: Over-reliance is the tendency for users or systems to treat model output as authoritative even when the model is uncertain or wrong. In AI governance, this is a delegation failure because the workflow assigns decision weight to output without enough review, confidence signalling, or containment.
• Prompt Boundary Integrity: Prompt boundary integrity is the ability of an LLM application to keep untrusted content, system instructions, retrieval results and hidden context from collapsing into one insecure instruction stream. It is a practical control property, not a model feature, and it determines whether guardrails can actually hold.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity and machine identity security. It helps security and identity practitioners translate emerging AI risk into controls their programmes can enforce.