Any-depth alignment exposes a new failure mode in LLM safety

At a glance

What this is: VirtueAI's paper argues that LLM safety controls can fail after generation has begun, and that reintroducing alignment tokens mid-stream can recover refusals.

Why it matters: This matters because AI safety and governance teams need controls that operate during inference, not only at prompt intake, especially as LLMs are embedded in agentic workflows.

By the numbers:

• ADA reduces the average success rate of prominent adversarial prompt attacks to below 3%.

Context

LLM safety controls are often evaluated as if refusal is a one-time decision at the start of a turn. This paper argues that the real governance gap appears during generation, where an attacker can steer the model past its initial refusal boundary and into compliant harmful output.

For identity and AI security teams, the relevant problem is not just prompt filtering but control persistence across the inference session. If a model can be nudged from safe to unsafe output mid-stream, then agentic workflows that rely on LLM output need monitoring, interruption points, and policy enforcement during generation, not only before it starts.

Key questions

Q: What breaks when LLM refusal only works at the start of a response?

A: The model can remain unsafe once a harmful continuation is underway, because the original refusal state may not survive deeper into generation. That creates a control gap between prompt-time filtering and runtime safety. For agentic systems, the danger is bigger because unsafe output can be handed directly to tools, memory, or downstream automation.

Q: Why do harmful prefixes and assistant-prefill attacks increase LLM risk?

A: They place the model into a generation trajectory where the initial refusal boundary is already bypassed. Once the model is continuing an unsafe path, conventional guardrails that only inspect the start of the turn are too late. The practical issue is not just content moderation, but preserving safety state across the full response.

Q: How can security teams tell whether an LLM is actually safe during generation?

A: They should test refusal durability at multiple depths, including long harmful prefixes, mid-response interruptions, and resumption cases. If safety collapses after the first few tokens, the control is shallow and unsuitable for workflows that depend on reliable denial behavior. The right signal is consistent refusal across the entire generation path.

Q: Should organisations treat LLM safety as a model issue or a runtime governance issue?

A: Both, but runtime governance is the decisive layer when the model is already in production. A model may be aligned in isolation and still fail once generation begins inside an agentic workflow. That is why policy enforcement, monitoring, and intervention points have to sit inside the execution path, not only around the prompt.

Technical breakdown

Why shallow refusal training leaves a generation-depth gap

Shallow alignment is a training side effect, not a guarantee of continuous safety. Many models learn strong refusal behavior when the assistant is expected to begin with a denial, but that prior weakens once the model is already producing content. The result is a generation-depth gap: safety is strongest at the first token of the assistant response and becomes less reliable as the continuation unfolds. This matters because attackers do not need to break the model outright if they can get it to continue from a permissive state.

Practical implication: Treat refusal as stateful behavior that can degrade during generation, not as a fixed property of the model.

How Any-Depth Alignment reuses assistant header tokens mid-stream

Any-Depth Alignment works by reintroducing assistant header tokens during generation. The paper's claim is that these tokens carry strong alignment priors because shallow-refusal training repeatedly ties them to safe response behavior. By injecting them again mid-stream, the method pushes the model to reassess harmfulness at arbitrary depths instead of only at the start of a turn. Importantly, this is an inference-time defense, so it does not require changing base model weights or retraining the model itself.

Practical implication: Use inference-time policy resets where model retraining is impractical, especially for shared or fast-moving LLM deployments.

Why the defence matters for prompt attacks and agentic workflows

The paper reports that Any-Depth Alignment holds up against adversarial prefill attacks spanning dozens to thousands of tokens and lowers the success of GCG, AutoDAN, PAIR, and TAP to below 3%. That is a meaningful operational signal because many attack methods rely on forcing the model into a harmful trajectory after the conversation has already begun. For agentic systems, the lesson is broader: if an LLM can be manipulated mid-generation, then downstream tool calls, memory writes, and delegated actions inherit that weakness.

Practical implication: Bind generation-time safety checks to any workflow where model output can trigger tools, memory updates, or autonomous actions.

Threat narrative

Attacker objective: The attacker wants to make the model continue unsafe generation after the point where it should have refused.

1. Entry occurs through adversarial prompting or harmful assistant-prefill content that places the model into a risky generation trajectory.
2. Credential-like trust in the current assistant state is abused as the model continues generating, allowing the attack to outlast the initial refusal boundary.
3. The attacker sustains the harmful continuation long enough to bypass safety controls and elicit unsafe output or prompt-attack success.
4. Impact is the production of harmful model output that can feed agentic workflows, tool use, or downstream decision logic.

Breaches seen in the wild

• McKinsey AI platform breach: McKinsey AI platform hack exposed 46M chats and sensitive data.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Any-depth safety is now a governance requirement, not a model nicety. The paper shows that refusal quality is depth-dependent, which means safety cannot be assessed only at the first response token or prompt ingress. For AI governance teams, that changes the unit of control from prompt screening to runtime assurance, especially in systems that chain LLM output into tools or policies. The practitioner conclusion is simple: safety must persist for the full generation path.

Generation-time control exposes a new AI governance debt. Many programmes assume that if a model is aligned at the start of a turn, the rest of the output inherits that state. This paper breaks that assumption by showing that alignment can be reactivated mid-stream through a simple inference-time mechanism. For organisations building agentic workflows, the result is a sharper boundary between model safety and system safety, and the system boundary is where governance has to live.

Inference-time resets are the right design pattern when retraining is not the answer. Any-Depth Alignment is notable because it addresses a failure mode without changing base weights, which is relevant for model fleets, shared services, and third-party models. That makes the control pattern attractive for environments where model ownership is limited but operational responsibility is not. The practitioner conclusion is to treat generation-time intervention as a first-class safety control, not an emergency patch.

AI agent oversight must include mid-generation intervention points. If a model can recover refusal at arbitrary depth, then the security programme should also be able to interrupt, re-evaluate, or constrain generation at arbitrary depth. This is where LLM safety intersects with agentic AI governance: once model output can drive action, a late-stage unsafe token is no longer just a content problem. The practitioner conclusion is to design for runtime containment, not just pre-prompt filtering.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Generation-depth controls are the next practical boundary for AI safety. Teams that rely on LLMs for summarisation, coding, or action-taking should assume that prompt-time refusal is insufficient once the model enters a long continuation. The operational question is whether the system can reassert policy after unsafe drift has begun, not whether the first answer looked safe.

Mid-stream refusal recovery is a useful design pattern for agentic AI governance. When models can call tools or influence decisions, safety needs to be enforceable at the point where output becomes action. That pushes AI governance toward runtime intervention, staged verification, and stronger observability over generation states.

For practitioners

• Map safety controls to generation depth Test whether refusal holds after harmful continuations begin, not only at prompt entry, and record where the model loses safety state.
• Add mid-stream policy resets to inference pipelines Insert a control point that can reassert safety instructions or headers during long generations, especially for agentic and tool-using systems.
• Evaluate prompt-attack resistance under adversarial prefill Benchmark long harmful prefixes against the models you actually deploy, including cases where the attack spans dozens or thousands of tokens.
• Constrain tool execution to post-refusal verification Block downstream actions until the model output has passed a fresh safety check at the current generation state, not the initial prompt state.

Key takeaways

• LLM safety can degrade after generation begins, which leaves a gap between initial refusal and runtime behavior.
• Any-Depth Alignment shows that mid-stream reassertion of safety priors can restore refusals without changing base model weights.
• For agentic systems, the governance problem is no longer just what the model says first, but whether safety still holds before output becomes action.

Key terms

• Shallow Alignment: A safety pattern where a model is most likely to refuse harmful requests near the start of a response, but loses that resistance once generation has already moved into an unsafe path. It is a sequence-sensitive behavior, not a guarantee of safety across the full output.
• Generation-depth Gap: The difference between how safe a model looks at the first tokens of a reply and how safe it remains later in the same generation. This gap matters because adversaries can exploit continuation state, not just the initial prompt.
• Inference-time Defence: A control applied while the model is generating output, rather than during training or fine-tuning. It is useful when the base model cannot be changed, but the deployment still needs runtime safety enforcement and policy resets.
• Prompt Attack: An attempt to influence an AI system by shaping the input in ways that bypass safeguards, change behaviour, or reveal protected information. In connected systems, the risk is not limited to wording, because the manipulated response can trigger tools, access data, or initiate downstream actions.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and secrets management for practitioners building controlled AI systems. It helps security teams connect identity governance to the operational realities of modern AI deployments.