The difference between how safe a model looks at the first tokens of a reply and how safe it remains later in the same generation. This gap matters because adversaries can exploit continuation state, not just the initial prompt.
What the generation-depth gap means
The generation-depth gap describes a failure mode in which a model appears well-behaved at the start of a response, then becomes less safe or more permissive as generation continues. The key issue is that safety can drift within a single completion, not only across separate prompts.
This matters because a short safety check on the opening tokens can miss unsafe continuation behaviour later in the same answer. The gap can show up as gradual policy erosion, delayed disclosure, or a response that starts with restraint and then shifts once the model has more self-generated context to work from.
Why the gap exists
The gap is usually a property of sequence generation rather than a single broken rule. Early tokens are often shaped by the prompt and the model’s initial alignment state, but later tokens are influenced by its own prior output, which can create path dependence and weaken the original constraint signal.
That makes continuation state important. Once the model has produced a plausible opening, later tokens may follow a momentum of their own, especially if the unsafe route is not re-evaluated token by token or if the safety layer is only strongest at the first-step decision point.
How it affects evaluation and design
For safety testing, the generation-depth gap means a model should not be judged only on first-token compliance or on whether it passes a brief prompt-level filter. Evaluators need to inspect the whole completion, including long answers, multi-step reasoning, and cases where the model is encouraged to continue after an initially safe start.
For system design, the gap suggests that guardrails should be applied throughout the generation path, not just before output begins. It also means refusal behaviour, policy checks, and post-generation filtering need to account for late-emerging unsafe content that was not visible at the start of the response.
Where it shows up in practice
The generation-depth gap is easiest to observe in answers that unfold over several sentences or steps. A model may start with a compliant summary, then drift into unsupported instructions, policy-bypassing details, or other content that would have been blocked if judged in isolation at the beginning.
That makes the term especially useful in red teaming, safety auditing, and benchmark design, where the goal is to measure whether alignment persists across the full generation rather than at a single checkpoint. It is a reminder that apparent safety at the first few tokens does not guarantee safe behaviour at token 50 or token 500.
Risk and Threat Considerations
The generation-depth gap creates a practical safety exposure because adversaries can probe for weak continuation behaviour after an initially harmless start. If defenders only evaluate the beginning of a response, they may miss unsafe escalation later in the same generation.
Failure mechanism: The model’s later tokens inherit and amplify its own generated context, which can weaken the original safety constraint and allow unsafe continuation even when the opening looked compliant.
Impact: Unsafe details, policy-violating content, or misleading outputs can emerge deeper in the reply, reducing trust in the model and undermining prompt-time safety checks.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | Govern | Covers AI risk governance for evaluating safety drift across generation. |
| Recommendation — Establish continuous AI risk monitoring across the full generation path. | ||
| NIST SP 800-53 Rev 5 | SI-4 — System Monitoring | Monitoring is needed to detect unsafe continuation behaviour in outputs. |
| AU-6 — Audit Record Review, Analysis, and Reporting | Reviewing records supports post-hoc analysis of where responses become unsafe. | |
| Recommendation — Monitor outputs for late-emerging unsafe content and anomalous response drift. Review generation logs to pinpoint when safety degradation begins. | ||
| OWASP Agentic AI Top 10 | ASI06 — Memory & Context Poisoning | Context-dependent agent safety can degrade as the model continues generating. |
| Recommendation — Test whether accumulated context causes later-token safety drift. | ||
| MITRE ATLAS | Adversarial machine learning techniques | Adversarial prompting and continuation attacks fit AI threat modelling. |
| Recommendation — Model continuation-based prompt attacks in your AI threat tests. | ||
Practitioner Guidance
What to watch for: Treat long-form outputs as a continuous safety surface, not a one-time decision. Test whether the model remains consistent after multiple turns, extended completions, and self-referential continuation, because the gap often appears only after the initial safe segment.
Common misunderstanding: A safe opening is not proof of a safe response. Practitioners should evaluate whether policy enforcement, refusal logic, and output filtering are robust across the full generation path, especially in systems where users can elicit extended continuations.