LLM resilience testing shows why prompt injection still matters

At a glance

What this is: Cymulate describes LLM resilience testing that simulates prompt injection and jailbreak attacks, finding that content filters alone do not reliably stop unsafe model behaviour.

Why it matters: IAM, AI security, and platform teams need to treat LLM prompts, tool use, and execution rights as governed attack surfaces, not just model output problems.

Context

Prompt injection is an LLM control failure, not just a content moderation problem. When instructions, system prompts, and user input share the same execution path, an attacker can reshape model behaviour without needing to breach the underlying platform.

The identity angle matters because copilots, agents, and autonomous tools often act on behalf of users or services. That turns the model into a privilege-bearing runtime, where authentication, authorisation, and action boundaries have to be assessed alongside model safety.

This article is about a testing method for LLM resilience, and that makes the governance question more important than the demo itself: can organisations prove how their models behave when adversarial prompts, obfuscation, and tool execution are combined?

Key questions

Q: What breaks when LLM prompt injection is not separated from trusted instructions?

A: When trusted policy and untrusted user content share the same instruction stream, attackers can overwrite or steer model behaviour with obfuscated prompts, roleplay, or reverse instructions. The result is not just bad text generation. It is a loss of control over tool use, downstream actions, and policy enforcement.

Q: Why do AI copilots and agents make prompt injection risk worse?

A: They widen the impact of a successful injection. A plain chatbot may only produce unsafe text, but a connected AI system can search data, invoke tools, write records, or trigger workflows. Once the model can act, the attacker is no longer only manipulating output. They are steering privileged behaviour through the model.

Q: What do organisations get wrong about LLM safety filters?

A: Many organisations assume a filter that blocks obvious harmful wording is enough. Tokenisation confusion and policy simulation show that attackers can evade literal checks while preserving the same intent. Effective control requires semantic testing, adversarial red-teaming, and isolation before any output can drive action.

Q: How should security teams test LLMs for jailbreak risk before production?

A: Test the model with both single-turn and multi-turn adversarial prompts, plus indirect injection from the same content channels your application will ingest. Score success by harmful output category, not just refusal rate, and repeat the tests after every prompt, model, or tool-chain change so you can see whether the attack surface is expanding or shrinking.

Technical breakdown

Why prompt injection works against shared instruction streams

LLMs typically receive a system prompt, user prompt, and sometimes tool instructions in one conversational flow. Prompt injection exploits that shared context by inserting adversarial instructions that compete with or overwrite the intended behaviour. Jailbreaks add obfuscation, roleplay, or multi-step framing to reduce the chance that safety layers flag the payload. The core weakness is not simply weak moderation, but ambiguous instruction hierarchy and insufficient separation between trusted policy and untrusted input.

Practical implication: treat prompt boundaries as a control design problem, not a moderation setting.

How digital-twin testing exposes LLM failure modes safely

A digital twin gives defenders a live replica of the target LLM environment so they can test prompt injection, output abuse, and tool misuse without touching production. Cymulate describes this as a repeatable testbed with YAML-defined scenarios, verdict parsing, and structured logging. That matters because many LLM failures only appear under adversarial pressure or when prompts are mutated in ways ordinary testing never exercises. Safe replication is essential when models can generate code, trigger actions, or reach connected tools.

Practical implication: validate LLM controls in a non-production twin before exposing models to real users or automation.

Why filtering alone does not contain agentic LLM risk

Content filters can block obvious disallowed text, but they do not necessarily stop indirect execution, contextual camouflage, or payloads hidden inside narrative or code-like wrappers. Once an LLM can act as a copilot or agent, the security question shifts from text output to action authority. If the model can trigger scripts, call services, or steer downstream workflows, then prompt injection becomes a privilege abuse problem as much as a model safety issue. That is where identity, tool scope, and execution controls intersect.

Practical implication: pair filtering with tool authorisation, output validation, and execution containment.

Threat narrative

Attacker objective: The attacker’s objective is to steer the LLM into bypassing safeguards and carrying out outputs or actions that the operator did not intend.

1. Entry begins when an attacker submits a crafted prompt, roleplay, or obfuscated instruction that blends into normal LLM interaction.
2. Credential and instruction abuse follows when the model accepts the injected content as a higher-priority directive and produces unsafe output or action.
3. Escalation occurs when the model is allowed to call tools, generate code, or influence downstream workflows under that poisoned instruction set.
4. Impact is model misbehaviour, policy bypass, or unsafe execution that can leak instructions, trigger unapproved actions, or compromise connected systems.

Breaches seen in the wild

• EchoLeak (Microsoft 365 Copilot) 2025: A crafted email could make Microsoft 365 Copilot leak data from its context with no click, a zero-click prompt injection fixed as CVE-2025-32711.
• ForcedLeak (Salesforce Agentforce) 2025: A critical Agentforce flaw let attackers steer an AI agent through a Web-to-Lead form to leak CRM data via an expired allowlisted domain.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Prompt injection is now a governance issue, not a novelty attack. Once LLMs sit inside copilots, agents, and workflow automation, the attack target is no longer just model output. It is the model's authority to interpret instructions and trigger actions. Security teams should treat prompt hierarchy, tool scope, and output handling as part of the control plane.

Instruction hierarchy failure: is the named concept this article sharpens. The underlying problem is that trusted and untrusted instructions are still too often processed in the same conversational stream. When that happens, obfuscation and roleplay can defeat naive filtering even when the platform blocks obvious abuse. Practitioners should assume the control gap sits at instruction separation, not only at content moderation.

Agentic LLMs inherit the identity problem from automation. When a model can act on behalf of a user, service, or workflow, the relevant question becomes what it is authorised to do, not only what it can say. That makes identity, delegated authority, and execution containment inseparable from AI safety. Organisations should align AI governance with access governance, not treat them as separate programmes.

Production-safe adversarial testing is becoming a baseline requirement. The value of a digital twin is that it surfaces failures before the real environment is exposed. This is especially important for organisations evaluating copilots or autonomous tools in regulated or high-impact workflows. The practitioner conclusion is clear: if you cannot simulate prompt abuse safely, you do not yet understand your real operating risk.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Agentic AI Security Guide

What this signals

LLM prompt safety is converging with identity governance. Once a model can invoke tools or act on behalf of a user, the real question is whether its authority is bounded. That is why prompt isolation, tool scoping, and approval logic need to be governed like privileged access, not treated as model tuning.

Instruction hierarchy failure is the operational pattern practitioners should now watch for. If adversarial prompts can still reshape behaviour after filtering, then the control objective has to move upstream to trusted context, execution gating, and post-generation validation.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026. That scale reinforces a simple point: AI security problems rarely stay inside the model boundary.

For practitioners

• Define prompt trust boundaries Separate system instructions, user input, and tool directives so the model cannot treat untrusted content as policy.
• Test against obfuscated jailbreaks Run scenarios that use roleplay, reverse instructions, token stuffing, and narrative camouflage to confirm controls still hold.
• Constrain tool execution rights Limit what copilots and agents can invoke, and require explicit approval for actions that change data or state.
• Validate outputs before execution Inspect generated code, commands, or API calls before they reach downstream systems or automation runners.
• Use a digital twin for adversarial testing Mirror the target LLM environment in a safe testbed so prompt injection and jailbreak cases can be exercised without production risk.

Key takeaways

• Prompt injection remains a practical risk because LLMs still struggle to distinguish trusted policy from adversarial user input when both share the same context.
• The article shows that safer testing needs production-like replicas, not just static policy checks, because many failures only appear under obfuscation and chained prompts.
• For practitioners, the decisive controls are instruction separation, scoped tool execution, and output validation before any generated action reaches downstream systems.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Jailbreak: A prompt or input transformation that bypasses a language model’s safety restrictions and causes it to produce output it would normally refuse. In operational settings, a jailbreak matters because the model may be embedded in a workflow, making the bypass a pathway to broader misuse, not just a bad response.
• Digital Twin: A digital twin is a high-fidelity virtual representation of a physical system, environment, or process. In security and identity work, it becomes sensitive when it is used to generate data, validate models, or control real-world decisions, because access to the twin can expose operational knowledge and deployment paths.
• Instruction Hierarchy: Instruction hierarchy is the order of authority a model applies when interpreting context, system prompts, role messages, and user input. When attackers can influence that hierarchy through templates or wrappers, they can steer behaviour without needing to change the model itself.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It helps practitioners connect identity control to the broader security decisions that matter in modern automation.