AI explainability is creating new jailbreak paths for LLM security

At a glance

What this is: CyberArk’s research shows how AI explainability can help derive new jailbreak techniques for LLMs, including variants that bypass safeguards in open- and closed-source models.

Why it matters: IAM and security teams need to treat LLM output as untrusted because when models influence application behaviour, jailbreak risk becomes a governance and control-design issue, not only a model-tuning issue.

Context

AI explainability is the practice of inspecting how a model reaches its outputs, including which internal signals change as prompts change. In this article, CyberArk applies that idea to jailbreak research, showing how visibility into LLM internals can be turned into a way to find weak spots in model safeguards.

The governance gap is not that models can be attacked in abstract. It is that organisations increasingly embed LLMs into workflows where output can influence decisions, routing, or downstream actions. Once that happens, a bypass of the model's refusal behaviour becomes a control failure for the surrounding application, not just a model issue.

Key questions

Q: How should security teams layer defenses around LLM applications when jailbreaks keep succeeding?

A: Treat model alignment as one control, not the control. Security teams should combine prompt filtering, output moderation, abuse detection, rate limits, and access controls around the application path, then test those controls against multi-turn and adaptive jailbreaks. The goal is to reduce the chance that any single bypass exposes sensitive outputs, unsafe actions, or downstream workflow abuse.

Q: Why do jailbreaks become a serious security risk when AI agents have tool access?

A: Jailbreaks matter more in tool-enabled environments because the model is no longer just generating unsafe text. It can execute file, email, database, and API actions using legitimate credentials. That turns a prompt-level bypass into an operational security incident, where a single successful manipulation can trigger exfiltration, unauthorized communications, record changes, or code execution before human review catches it.

Q: What signs show that a workflow is trusting LLM output too much?

A: Warning signs include prompts that can trigger downstream actions, responses that are executed without review, and controls that assume the model will always refuse unsafe requests. If a jailbreak can change a business process, the workflow is over-trusting the model.

Q: Should organisations use explainability findings to improve model safety?

A: Yes, but only as one input to a layered control design. Explainability can help identify weak layers or refusal patterns, yet it should inform validation, gating and monitoring rather than becoming the trust basis for the model itself.

Technical breakdown

How adversarial AI explainability maps jailbreak behaviour

Adversarial AI explainability combines internal model inspection with offensive testing. Instead of treating a jailbreak as only a prompt trick, the method compares benign prompts, harmful prompts and successful bypasses to identify neurons, layers or paths that react differently. The article describes this as finding safety-critical neurons, safety-critical layers and weak layers, then using those signals to shape new jailbreak variants. The technical point is not that explainability is inherently defensive or offensive. It is that visibility into internal activations can reveal which parts of the model appear to anchor refusal behaviour and which parts appear easier to perturb.

Practical implication: treat model introspection results as attack intelligence, not as evidence that the model can enforce policy on its own.

Why jailbreaks become more serious in agentic AI workflows

The risk changes when an LLM does more than answer questions. In an agentic workflow, the model may help set goals, choose actions or influence application behaviour, so a successful jailbreak can affect more than a single response. That shifts the blast radius from content generation to operational decision flow. The article is explicit that LLM output is not a security control, which means any workflow that trusts the model to validate intent, enforce policy or gate action is building on a fragile assumption. The core issue is governance of how the output is used, not only how the model is aligned.

Practical implication: separate model generation from policy enforcement and require independent validation before any model output can drive action.

Why layered safeguards matter more than model alignment alone

The post distinguishes between model alignment and external guardrails such as input filtering, output moderation, human review and access controls. Alignment changes the model's response tendencies, but the research argues that jailbreaks can still surface through encoding tricks, indirect prompts or other manipulations of the input space. That means safety cannot depend on the model internalising policy correctly every time. In practice, the surrounding system has to assume the model may produce unsafe or misleading output and then constrain what the application will accept, store or execute based on that output.

Practical implication: design a layered control stack that validates, filters and limits model output before it affects business logic.

Threat narrative

Attacker objective: The objective is to bypass safety restrictions and coerce the model into producing or enabling outputs that the surrounding system should have blocked.

1. Entry occurs through indirect prompting, encoding tricks or other prompt transformations that reshape the input without changing the harmful intent.
2. Credential-style access is replaced by model-state access: the attacker influences internal activations, safety layers or decision paths until refusal behaviour weakens.
3. Escalation happens when the bypassed model is embedded in a workflow that trusts its output, allowing manipulated content to steer application behaviour.
4. Impact is the generation of harmful responses or the compromise of downstream decision-making when the model is treated as an enforcement layer.

Breaches seen in the wild

• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.
• McKinsey AI platform breach: McKinsey AI platform hack exposed 46M chats and sensitive data.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM output cannot be treated as a security control: CyberArk's research reinforces a basic governance boundary that many AI programmes still blur. A model may assist with policy decisions, but it cannot safely act as the policy enforcement point. The implication is that approval, validation and access decisions must live outside the model, or jailbreaks become control-plane failures.

Adversarial AI explainability creates a runtime trust problem, not just a research problem: Once internal model signals can be inspected, they can also be used to optimise attacks. That means explainability has dual use in exactly the place security teams care most about, where model behaviour meets operational trust. The implication is that explainability outputs need governance, not unrestricted operationalisation.

Agentic workflows amplify jailbreak impact because action follows language: The article shows why model compromise matters more when LLMs influence system behaviour. In those designs, a successful jailbreak is not only a bad answer; it is a path to action steering. The implication is that agentic AI governance has to assume prompt manipulation can become privilege manipulation through application logic.

Identity governance must extend to model-mediated authority: The article points toward a broader control problem in which systems grant effective power to model output without assigning that power an accountable identity, privilege boundary or review path. That is a governance weakness, not an AI novelty. The implication is that any workflow using LLM decisions needs explicit accountability for who or what is authorised to trigger action.

Explainability turns jailbreak resistance into a measurable control surface: CyberArk's work suggests that internal activations, weak layers and refusal patterns can be used to build better detection and tuning strategies. That is valuable, but only if teams stop treating model safety as a binary property. The implication is that AI governance should measure where the model fails, then constrain the blast radius of those failures.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Threat Modelling AI Agents

What this signals

Adversarial AI explainability: the more visibility organisations gain into model internals, the more care they need around how that information is used. For security programmes, explainability should support testing and mitigation, but it should never be mistaken for proof that the model can safely enforce policy or authorise action.

When LLM output influences application behaviour, the control question shifts from model safety to decision integrity. That is why security and identity teams need to define where the model ends and where governed enforcement begins, especially in agentic workflows.

The practical boundary is simple: if a jailbreak can change what the system does, then the model is part of the attack surface, not the control surface. Programmes that keep that distinction clear will make better decisions about validation, gating and accountability.

For practitioners

• Separate model output from enforcement logic Require a non-LLM control layer to decide whether a prompt, response or inferred action is allowed to proceed. Use the model for assistance, not for authorisation, policy enforcement or final approval.
• Instrument jailbreak-style prompt testing Test harmful prompts, encoding transformations and indirect prompt variants against your own workflows to see where refusal behaviour breaks and where downstream systems still trust the output.
• Constrain agentic workflows with explicit approval gates If an LLM can influence routing, execution or permissions, require deterministic gates outside the model before any action is taken. Do not allow model confidence or refusal patterns to substitute for control checks.
• Review how explainability data is operationalised Treat neuron, layer and activation analysis as sensitive research material. Limit who can access it, document how it informs mitigation, and avoid using it as a standalone assurance claim.

Key takeaways

• AI explainability can help reveal why some jailbreaks work, but the same insight can also make attacks more effective if it is not governed.
• The main risk appears when model output is allowed to influence downstream actions, because jailbreak success then becomes a workflow-control problem.
• Teams should keep enforcement outside the model, add independent validation, and treat LLM behaviour as untrusted input in agentic designs.

Key terms

• Adversarial AI Explainability: A research approach that uses model-introspection and behavioural analysis to understand how adversarial prompts affect an LLM. It is not a security control by itself. Its value is in revealing fragile model behaviours that defenders can test, monitor, and harden before those behaviours are exploited in production.
• Jailbreak: A prompt or input transformation that bypasses a language model’s safety restrictions and causes it to produce output it would normally refuse. In operational settings, a jailbreak matters because the model may be embedded in a workflow, making the bypass a pathway to broader misuse, not just a bad response.
• Agentic workflow: An agentic workflow is a sequence of tasks executed by an AI agent with some level of tool access and decision authority. In security terms, the workflow matters because it can span multiple systems, identities, and permissions, which makes attribution and revocation harder than with ordinary automation.
• Guardrail Layering: Guardrail layering is the practice of combining multiple independent controls so that one failure does not expose the full system. In AI security, that usually means pairing cloud configuration controls, model behaviour checks and identity restrictions across the same workflow.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.