Any-depth alignment exposes a new failure mode in LLM safety

At a glance

What this is: This research shows that LLM refusal behavior can fail after a harmful response has already started, and that reinserting assistant-header tokens mid-stream can recover safety checks without changing model weights.

Why it matters: It matters because teams building AI assistants and agents need controls that interrupt unsafe generation in-flight, not just pre-prompt filters or static policy layers, especially where LLM outputs can trigger tools or downstream actions.

By the numbers:

• ADA secures a near-100% refusal rate against adversarial prefill attacks ranging from dozens to thousands of tokens.
• 3%.

👉 Read VirtueAI's paper on Any-Depth Alignment for LLM safety

Context

LLM safety controls often assume the risky prompt appears before the model starts responding. This paper shows that assumption is too narrow: once harmful continuation is underway, shallow refusal patterns can collapse, which creates a governance gap for prompt injection, adversarial prefill, and agent workflows that generate long outputs.

For IAM and NHI teams, the relevant issue is not just model behavior but execution boundary control. If an LLM or agent can continue generating after the unsafe path has begun, policy checks need to reassert identity, intent, and authorization during the session, not only at initiation.

That makes this a model-governance problem with identity consequences. The starting position of many AI deployments is typical, but the failure mode becomes more dangerous as models are embedded into autonomous or semi-autonomous workflows.

Key questions

Q: How should security teams test whether LLM safety controls still work after harmful generation starts?

A: Test across long contexts, adversarial continuation, and assistant-prefill scenarios, not just first-turn prompts. The key is to measure whether refusal behavior survives after the model has already entered an unsafe trajectory. Include tool-use and multi-turn workflows, because those are where shallow alignment is most likely to fail.

Q: Why do shallow refusal controls fail in long LLM interactions?

A: They often depend on prompt position and sequence context, so the model’s refusal tendency is strongest at the start of a response and weaker after harmful continuation begins. In long interactions, the unsafe path can outrun the safety check unless runtime controls reassert policy during generation.

Q: What do security teams get wrong about LLM safety at runtime?

A: They assume a single moderation step at prompt ingress is enough. In practice, risky behavior can emerge later in generation, especially when the model is producing code, tool instructions, or delegated actions. Runtime safety has to be continuous, observable, and tied to execution boundaries.

Q: How should organisations govern AI agents that rely on long-form LLM generation?

A: Treat the model as a component inside a governed execution chain, not as the final decision-maker. Require re-evaluation before tool calls, log refusal depth, and separate output generation from authority to act. That approach reduces the chance that unsafe text becomes unsafe execution.

Technical breakdown

Why shallow alignment breaks after harmful continuation begins

The paper describes shallow alignment as a refusal tendency that is strongest at the start of an assistant turn and weaker once the model has already entered a harmful continuation. That matters because many safety mechanisms are effectively front-loaded. If an attacker can push the generation path past the initial refusal boundary, the model may continue producing unsafe content even when the underlying preference for refusal still exists. This is not the same as full jailbreak resistance. It is a depth problem in generation control, where the sequence context changes the model’s behavior before the safety check can reassert itself.

Practical implication: evaluate safety controls at multiple generation depths, not only on first-turn prompts.

How assistant-header tokens can restore refusal priors mid-stream

ADA works by reintroducing assistant-header tokens during generation. The paper’s core observation is that repeated shallow-refusal training concentrates alignment signals in those header tokens, so reusing them later can prompt the model to reassess harmfulness. In practice, this is an inference-time intervention, not a weight update. That makes it attractive for environments where model modification is impractical. The mechanism is useful because it targets the model’s own refusal prior rather than trying to layer on an external classifier after the fact.

Practical implication: test whether runtime prompt structure can re-trigger refusal behavior before relying on external moderation alone.

What this means for agentic AI safety boundaries

For agentic systems, the lesson is that a model’s safety posture must hold across the whole generation span, including long outputs, tool-planning text, and delegated instruction chains. A system that appears safe at turn start can still drift into unsafe output if the model is not re-prompted or re-checked during the session. This shifts the conversation from single-shot prompt filtering to ongoing runtime control. For governance, the control question is whether the assistant can be forced to re-evaluate harmfulness before it emits actionable instructions or tool calls.

Practical implication: insert runtime re-evaluation points before tool use or long-form generation completes.

NHI Mgmt Group analysis

Shallow alignment is now a runtime governance problem, not a prompt-filtering problem. The paper shows that refusal behavior can disappear after harmful continuation starts, which means pre-prompt controls alone are insufficient. For AI security programmes, the control boundary must move into the generation loop where the model is actually producing risky content. That is especially important when an LLM sits behind tools, memory, or delegated actions, because unsafe text can become unsafe execution.

Any-Depth Alignment highlights a named concept we should track: the depth-of-generation trust gap. This is the mismatch between where safety policies are evaluated and where risky behavior actually emerges. If alignment only holds at the first token of a response, the organisation has a false sense of coverage. Practitioners should treat generation depth as a governance variable, not a model curiosity.

Inference-time safety can reduce dependence on base-model changes, but it does not replace policy design. ADA may improve refusal behavior without retraining, which is operationally attractive for mixed-model estates. But that also means teams must know which layers are responsible for re-evaluation, logging, and override decisions. In practice, the right question is whether runtime safeguards are observable and enforceable across the AI control plane.

Agentic AI programmes need controls that reassert intent during execution. If a model can keep generating after harmful content begins, then the safety problem extends to session management, tool invocation gates, and content provenance. This reinforces why OWASP Agentic AI and NIST AI RMF both matter for production deployments. Practitioners should align model safety with execution governance, not treat them as separate domains.

This research makes safety evaluation more demanding, not less. A defense that performs well on shallow prompts can still fail under adversarial continuation or prefill. That means red teams should test multi-stage prompts, long-context attacks, and assistant-prefill scenarios as standard practice. The operational conclusion is straightforward: measure refusal depth, not just refusal rate.

What this signals

Depth-aware refusal testing will become a baseline control for production AI programmes. Teams that only validate first-turn moderation will miss the failure mode this paper exposes, so red teams should add long-context and continuation tests to their standard validation set. For practitioners, that means measuring where safety breaks, not just whether it exists.

The practical governance lesson is that AI runtime controls need observability at the generation layer. If your programme cannot see when a model re-enters a harmful path, it cannot prove that its safeguards work under stress. That is where model governance, execution controls, and audit logging need to converge.

For practitioners

• Test refusal behavior at multiple generation depths Evaluate model responses against adversarial continuation, long prefill, and multi-turn escalation so you can see where refusal breaks after the initial token boundary. Include tool-call scenarios and long-form completions in your test set.
• Add runtime re-evaluation before tool use Require the assistant to re-check harmfulness before it emits tool instructions, code, or delegated tasks. Treat this as a control point in the AI execution path, not a post-processing filter.
• Instrument generation-depth telemetry Log where refusals occur, where harmful continuations begin, and whether re-prompting changes model behavior. That evidence helps distinguish a shallow safety win from a control that actually holds in production.
• Map safety controls to AI RMF and OWASP Agentic AI Document which policy, monitoring, and escalation controls govern prompt handling, model output review, and tool execution. Use that mapping to close gaps between model safety testing and production runtime governance.
• Separate model safety from execution authority Prevent a model from translating unsafe text into direct action unless a separate control confirms intent, context, and authorization. This matters most where LLMs can drive agents, APIs, or workflows.

Key takeaways

• LLM alignment can be shallow enough to fail after harmful continuation begins, which turns generation depth into a security concern.
• Any-Depth Alignment suggests runtime re-prompting can recover refusal behavior without changing model weights, but it still requires governed execution boundaries.
• AI security teams should test refusal depth, instrument generation telemetry, and separate output generation from the authority to act.

Key terms

• Shallow Alignment: A safety pattern where a model is most likely to refuse harmful requests near the start of a response, but loses that resistance once generation has already moved into an unsafe path. It is a sequence-sensitive behavior, not a guarantee of safety across the full output.
• Adversarial Prefill: A prompt attack technique that seeds the assistant with malicious continuation text so the model is nudged into unsafe generation before its refusal behavior can trigger. It exploits context dependence in the decoder rather than breaking the model directly.
• Generation Depth: The point in a model’s output where a response is evaluated, measured from the start of generation through later continuation. In safety terms, it is a control variable because models can behave differently before and after the initial refusal boundary.
• Runtime Re-evaluation: A control pattern that forces an AI system to reassess harmfulness during generation instead of only at prompt intake. It is useful when long outputs, agents, or delegated actions create risk after the first moderation check has passed.

What's in the full report

VirtueAI's full research covers the mechanism detail this post intentionally leaves for the source:

• Empirical attack testing across adversarial prefill lengths from dozens to thousands of tokens.
• Comparative refusal-rate results across open-source model families including Llama, Gemma, Mistral, Qwen, DeepSeek, and gpt-oss.
• The specific inference-time token insertion approach used to re-trigger refusal behavior.
• Benchmark notes on how the method performs after subsequent instruction tuning.

👉 VirtueAI's full paper covers the attack benchmarks, model families, and inference-time defense details.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and the control patterns that matter when AI systems need governed access. It helps practitioners connect identity discipline to AI runtime decisions across modern security programmes.