Long-Context Erosion

How long-context erosion changes model behaviour

Long-context erosion is not a simple “memory limit” problem. As sessions grow, the model can become less stable at weighting earlier instructions, which means the same prompt may produce different compliance, refusal, or prioritisation outcomes later in the conversation.

This matters because the degradation is often gradual rather than abrupt. A long session can still look coherent while the model quietly shifts attention toward recent tokens, repeated themes, or nearby phrasing, especially when the conversation keeps returning to a sensitive topic.

Why instruction following weakens as context grows

The core issue is attention distribution. With more tokens in play, the model must allocate limited attention across a larger working set, and important instructions can lose relative salience even when they remain technically present in the context window.

That can affect several behaviours at once, including refusal quality, adherence to system-style instructions, and consistency in how the model interprets user intent. Repetition, conversational drift, and long chains of partial instructions can all make the erosion more visible.

Where long-context erosion is operationally important

The effect is especially relevant in workflows that depend on sustained policy enforcement, careful red teaming, or high-consequence decision support. A model that performs well early in a session may become less reliable once the interaction accumulates many turns, retrieved passages, or nested tasks.

For agentic or tool-using systems, the concern is broader than text quality alone. If the model begins to underweight earlier constraints, it can mis-handle routing, over-accept later instructions, or lose track of boundaries that were meant to persist across the session.

Because context handling is a known weakness class in AI security, adversarial technique references such as the MITRE ATLAS adversarial AI threat matrix help frame how attention manipulation, context poisoning, and related behaviours are studied by defenders.

How teams reduce the impact of long-context erosion

Defensive design usually focuses on keeping critical instructions short, durable, and easy to reassert. Systems should avoid depending on a single early prompt to preserve policy, because long conversations can dilute that instruction even when it is still present in the visible history.

It also helps to treat long sessions as a reliability risk surface. Summarisation, context refresh, explicit state handoffs, and repeated constraint re-anchoring can reduce drift, but they should be validated rather than assumed to work in every model and deployment.

For runtime governance around model and tool interactions, the Model Context Protocol: Authorization specification is useful when sessions depend on disciplined token handling and bounded access paths. Broader AI risk controls in the NIST AI Risk Management Framework and the CSA MAESTRO agentic AI threat modeling framework also support structured review of drift, autonomy, and control failure modes.

Risk and Threat Considerations

Long-context erosion can create a practical security gap when the model becomes less reliable exactly where the session has already accumulated the most sensitive material, constraints, or adversarial influence. The risk is not just lower accuracy, but weakened refusal and boundary enforcement after the conversation has grown long enough to stress attention allocation.

Failure mechanism: Earlier instructions, safety boundaries, or task constraints lose salience relative to newer tokens, repeated content, or nearby sensitive framing, allowing later prompt material to steer behaviour more strongly than intended.

Impact: An attacker or careless user can exploit that drift to increase compliance with unsafe requests, degrade policy consistency, or cause the model to forget important session constraints during extended interactions.

Practitioner Guidance

What to watch for: Test long sessions, not just first-turn prompts. If refusal quality, instruction ordering, or task fidelity degrades as context length increases, treat that as a model-behaviour issue that needs its own evaluation rather than a one-off prompt failure.

Governance implication: Set acceptance criteria for long-context performance and revalidate them after model, prompt, retrieval, or toolchain changes. The control objective is stable behaviour across session length, not only good performance in short demonstrations.

Related resources from NHI Mgmt Group

• How do you decide between a long-context model and a faster assistant?
• What breaks when a coding agent loses context during a long session?
• How can organisations reduce risk when using long-context LLM APIs?
• How should security teams evaluate long-context AI systems before production use?