Intent drift and long-context safety: what practitioners need to watch

TL;DR: Across six models, adding intent drift produced no change in attack success rates, while topically relevant benign context still eroded guardrails and one frontier model failed up to 22% of the time at ~300k tokens, according to Manifold Security research. The signal is concentration on sensitive content, not drift or session length.

Editorial analysis by NHI Mgmt Group, based on content published by Manifold Security: “When a Conversation Wanders, Do an Agent's Guardrails Slip?”.

By the numbers:

• One frontier model's guardrail failed up to 22% of the time once ~300k tokens of harm-adjacent content had accumulated.
• A conservative drift alert fired 0.40 times per endpoint per day, or about 4,000 alerts a day across 10,000 endpoints.

Key questions

Q: What breaks when long AI sessions accumulate too much sensitive context?

A: The failure mode is not necessarily a dramatic prompt attack.

Q: Why does topical relevance matter more than intent drift in AI safety monitoring?

A: Because drift measures motion, while risk comes from accumulation.

Q: How should teams test whether refusal guards still work in long sessions?

A: Run the same harmful request at different context lengths while holding the attack constant, then compare refusal rates and unsafe completions.

Practitioner guidance

• Define sensitive-topic concentration thresholds Set internal thresholds for how much a session may accumulate around risky content before it triggers review, step-up controls, or reset logic.
• Separate drift detection from safety monitoring Do not treat topic switching as the main alert.
• Reset context before high-risk requests Force a fresh session or an isolated context window before instructions that could trigger unsafe completion, especially in long-lived support or coding workflows.

Bottom line: Topic drift alone did not explain the safety outcomes in this study, which means security teams should not rely on wandering conversations as a danger signal.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →