LLM attack testing shows prompt injection still breaks model guardrails

At a glance

What this is: This research tests four major LLMs against OWASP AITG-APP attack classes and finds that prompt injection, input leakage, and over-reliance remain the most reliable paths to failure.

Why it matters: It matters because security teams are now governing AI systems that can disclose data, follow malicious instructions, or produce unsafe outputs, which creates new control requirements for IAM, data protection, and AI governance programmes.

By the numbers:

• Pangea tested more than 1,000 unique payloads against each of four models to measure attack success rates across OWASP AITG-APP categories.
• Gemini 2.5 Flash showed a 69% fail rate for input leakage in the test set, making it one of the clearest weak spots in the comparison.
• Llama 4 Maverick recorded 76% fail rates for both over-reliance on AI and explainability and interpretability, indicating broad weakness in those scenarios.
• GPT-5's prompt injection fail rate was 4%, while indirect prompt injection was 6%, showing that even stronger models still need external controls.

👉 Read Pangea's analysis of LLM attack surface testing across four models

Context

Large language model attack surface testing is becoming a practical governance issue, not just a research exercise. As LLMs move into user-facing workflows, internal copilots, and decision support systems, the question is no longer whether a model can be tricked, but which attack classes produce repeatable failures and what controls sit around the model when that happens.

The article examines resilience across four models using more than 1,000 payloads mapped to OWASP AITG-APP categories. That makes the piece relevant to AI security, but also to identity and access teams because prompt injection, input leakage, and unsafe tool use can expose secrets, bypass intended instruction boundaries, or trigger actions that should have been constrained by governance.

The starting position described in this research is typical of the current market: strong claims about safety, but uneven resistance once adversarial prompts are applied. That gap is now familiar across enterprise AI deployments.

Key questions

Q: What breaks when prompt injection defenses are not in place for LLMs?

A: When prompt injection defenses are weak, attacker-controlled text can override the model's intended instructions, especially when the text arrives through retrieved content or external documents. The result is not always obvious compromise. It can be silent instruction hijacking, unsafe output, leaked context, or model behaviour that downstream systems trust too much.

Q: Why does clean core matter for identity and access governance?

A: Clean core matters because it changes where controls can live. When the SAP digital core is kept minimal, identity governance must operate through supported integrations and policy layers instead of bespoke code. That improves upgrade resilience, but only if IAM and GRC teams redesign controls for portability rather than assuming legacy extensions will carry forward.

Q: How do security teams know whether an LLM is operating safely?

A: Look for evidence that the model’s read scope, tool scope, and output handling are all bounded and reviewed. If the system can only act on approved data, only invoke approved tools, and logs those actions clearly, governance is working. If any of those are missing, the model is outside control.

Q: Who is accountable when an AI system makes a harmful decision?

A: Accountability should follow the identity chain that authorized, configured, or triggered the action, including the human owner, the platform team, and any delegated agent or tool account. If the organisation cannot name that chain, the governance model is too weak for regulated AI use.

Technical breakdown

Prompt injection and indirect prompt injection in LLMs

Prompt injection works when attacker-controlled text is treated as higher-priority instruction content than the system's intended policy or task boundary. Indirect prompt injection is more dangerous because the malicious instruction arrives through retrieved content, external documents, or tool outputs rather than a direct user prompt. In both cases, the model is not being "hacked" in the traditional exploit sense. It is being steered into behaviour the surrounding application did not intend, which exposes a governance failure in instruction hierarchy and context handling.

Practical implication: isolate untrusted content from instruction channels and test every retrieval or tool path for prompt injection resistance.

Input leakage, system prompt leakage, and context retention

Input leakage occurs when the model reproduces or reveals sensitive information that was placed in the prompt, conversation, or context window. System prompt leakage is a narrower case where hidden instructions, guardrails, or policy content are exposed back to the user. These failures matter because they show that the context window can become an exfiltration surface, especially when applications pass secrets, tokens, internal prompts, or business-sensitive text into model context without strict segregation.

Practical implication: remove secrets and privileged instructions from model context wherever possible, and treat prompt content as potentially recoverable data.

Over-reliance on AI and unsafe output governance

Over-reliance failures happen when a model produces advice or actions with too much confidence and too little qualification, even when the task should trigger uncertainty or escalation. This is not only a model-quality problem. It becomes a governance problem when users or downstream systems treat the output as authoritative enough to approve access, make operational decisions, or execute changes. In enterprise settings, the risk increases when AI output is connected to workflows that can affect identity, privilege, or data handling without human validation.

Practical implication: add approval gates, confidence thresholds, and human review for AI outputs that can influence access, secrets, or sensitive decisions.

Threat narrative

Attacker objective: The attacker aims to steer the LLM into exposing sensitive context or producing unsafe outputs that can be used to compromise trust, data, or connected workflows.

1. Entry occurs through attacker-crafted prompts, indirect prompt injection content, or malicious retrieval material that enters the model's context.
2. Escalation happens when the model prioritises the hostile instruction over its intended policy, revealing context, following unsafe directions, or amplifying untrusted output.
3. Impact follows when sensitive data, unsafe advice, or downstream workflow actions are produced with enough confidence to affect users, access decisions, or operational controls.

NHI Mgmt Group analysis

Prompt injection is now an access-control problem, not just a content-safety problem. Once a model can be steered through untrusted text or indirect context, the issue becomes enforcement of boundaries between trusted policy and attacker-supplied instructions. That is why LLM security has to sit alongside IAM, PAM, and data governance, not only inside model tuning. Practitioners need to treat instruction paths as controlled surfaces, not as advisory text.

Input leakage creates a hidden exfiltration channel across the AI stack. The more business processes pass secrets, internal prompts, or operational data into model context, the more likely the system is to reproduce something it should never expose. This is a governance failure in context handling, not a failure of user education. Organisations should redesign integrations so that the model sees the minimum data necessary to answer the task.

Over-reliance on AI is a named control gap: human approval assumptions break when model confidence is mistaken for policy enforcement. The article's fail rates show that unsafe outputs do not require model collapse, only enough trust in the response to let it shape real decisions. In identity-heavy workflows, that can mean access approvals, policy guidance, or secrets handling with no effective challenge. Teams need to define where AI output is advisory and where it is prohibited from triggering action.

AI governance debt is accumulating faster than most enterprises can measure it. The article compares models, but the deeper lesson is that organisations are deploying multiple AI systems without a consistent way to test attack surface, monitor failure rates, or assign accountability. That makes every new model another governance liability unless it is wrapped in testing, logging, and approval controls. Practitioners should demand measurable assurance before scaling adoption.

Identity and AI security are converging at the context boundary. When models can reveal secrets, misuse connected tools, or produce instructions that drive privileged workflows, the identity layer becomes part of the AI attack surface. That intersection should push IAM and security architecture teams to review authentication, authorisation, and secrets exposure around LLM integrations as a single control plane.

What this signals

AI security programmes are moving from model evaluation to runtime governance. The key signal is not whether an LLM can pass a benchmark, but whether the surrounding controls can stop prompt injection, suppress leakage, and prevent unsafe outputs from becoming business actions.

Context boundary control: The practical problem is no longer isolated prompt security. It is the governance of everything the model can see, retrieve, and influence, which means IAM, secrets handling, and approval workflows have to be designed with the LLM in mind.

If your programme cannot measure attack success rates per model and per use case, it does not yet have a defensible AI security control plane. That is the point where pilot activity becomes operational risk.

For practitioners

• Test against adversarial prompt classes Run red-team scenarios that cover direct prompt injection, indirect prompt injection, input leakage, and over-reliance cases before any production rollout. Use the OWASP Agentic AI Top 10 as a test planning reference and record fail rates by model and by workflow.
• Remove secrets from model context Review every LLM integration for secrets, tokens, internal prompts, and privileged business data that are being passed into the context window. Replace those inputs with references, scoped retrieval, or server-side lookups so the model never sees material that should not be recoverable.
• Gate AI outputs before they affect access Require human approval or policy validation for any AI output that can influence identity decisions, privilege changes, ticket closure, or workflow execution. Do not let model confidence act as an authorisation signal.
• Instrument model failure telemetry Track attack success rates, unsafe output patterns, and prompt leakage incidents per model and per use case, then review them as operational security metrics alongside the rest of the control stack.

Key takeaways

• LLM resilience varies widely, but prompt injection and leakage remain repeatable attack paths across major models.
• Governance failures happen at the context boundary, where untrusted text, secrets, and workflow inputs meet model behaviour.
• Enterprises need measurable AI controls before they let model output influence access, data handling, or operational decisions.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Input Leakage: Input leakage is the unintended exposure of data that was supplied to a model, including prompts, hidden instructions, or sensitive context. It becomes a governance issue when applications pass secrets or privileged information into the model and assume the model will not reproduce them.
• Over-Reliance On AI: Over-reliance on AI occurs when users or systems trust model output too much, treating it as authoritative even when it should be challenged or reviewed. In security programmes, this can turn advisory output into an operational decision point without sufficient validation.
• Session Boundary: A session boundary is the point where a browser interaction starts and ends, along with the controls that prevent state from leaking between tasks. In NHI governance, it is the practical line that determines whether cookies, tokens, and form data remain confined to one approved workflow.

What's in the full report

Pangea's full research covers the operational detail this post intentionally leaves for the source:

• Raw test results by model and attack category for teams that want to compare failure patterns directly.
• Judge LLM methodology and scoring notes for practitioners assessing how the fail rates were derived.
• Category-by-category discussion of specific OWASP AITG-APP scenarios and where each model struggled most.
• Repository access for the underlying results data, useful for internal validation or reproduction.

👉 The full Pangea post includes the model-by-model fail rates, methodology, and raw results repository.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, workload identity, and agentic AI identity. It helps security and identity practitioners build the control thinking needed for modern AI-enabled environments.