The control breaks when compaction or eviction removes the rule that was supposed to constrain the agent. The model may still be the same, but the behaviour changes because the instruction is no longer present in the retained state. For high-consequence actions, that means prompt-based governance is too fragile to rely on as the only enforcement layer.
Why context-window policy fails as soon as the agent’s working set changes
Policies that exist only as instructions in the context window are vulnerable to compaction, truncation, retrieval drift, and eviction. Once the rule is no longer present in the retained state, the agent can continue with the same base model but a different behavioural boundary. The failure is not model corruption, it is state loss: the constraint has become optional because it is no longer continuously available to the decision process.
That fragility matters most when the action is consequential, because the agent has no durable enforcement layer to fall back on. A prompt can guide behaviour while it is remembered, but it cannot guarantee continuity across long tasks, chained tools, or resumed sessions. For that reason, prompt-only governance should be treated as a soft control, not as the control of record.
For a practical reference point on the surrounding agent-risk landscape, NHIMG’s Agentic AI Security Guide maps the wider control problem across inputs, memory, tools, orchestration and identity.
What changes when the agent loses the rule mid-task?
The most important change is that the agent may still appear consistent while silently operating under a narrower or altered instruction set. In long-running workflows, that can mean earlier constraints are forgotten, lower-priority instructions survive, or a later tool call executes without the safety condition that was supposed to govern it. The risk is especially sharp when the policy was meant to limit permissions, require review, or block specific actions.
This is why the problem is not merely “the agent forgot something.” The real issue is that the decision boundary has become dependent on ephemeral text instead of a durable policy mechanism. Once that boundary is lost, there is no guarantee that the next action, subtask, or delegated tool invocation will be evaluated against the same rule.
For memory-specific failure modes, AI Agent Memory Security Guide is useful because it treats context retention, memory isolation, and cross-session leakage as security problems rather than convenience features.
What a durable control layer has to do instead
Durable control must live outside the transient prompt and be enforced at the point where the action is requested or executed. That usually means externalized authorization, per-action policy checks, scoped credentials, approval gates for high-impact actions, and logging that survives session resets. The goal is to make the agent’s authority testable even when the prompt is incomplete or gone.
When policy is externalized, the model can still reason, but it cannot self-authorise beyond the guardrail. That separation is the key architectural difference. The prompt can explain intent, but the enforcement layer decides whether the action is permitted, which is exactly what breaks when the policy exists only in context.
For teams implementing that separation, AI Agent Authorisation Guide gives the least-privilege pattern, and Zero Trust for AI Agents explains why the request, principal, and action should be verified continuously rather than assumed from prior context.
Risk and Threat Considerations
Prompt-only governance creates a brittle control surface because attackers, workflow changes, or simple context compaction can remove the very instruction meant to constrain the agent. When that happens, the agent may continue with residual privileges or tool access that no longer has the intended supervisory condition attached.
Failure mechanism: The policy is stored in ephemeral state, so eviction, summarisation, or task branching drops the constraint before the risky action is completed. The agent then behaves according to the remaining context, not the original governance intent.
Impact: High-consequence actions can proceed without the safeguard that was supposed to bound them, which increases the chance of unauthorized tool use, unsafe delegation, and hard-to-audit failures.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and CSA MAESTRO address the attack and risk surface, while NIST AI RMF and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Context-loss turns prompt policy into weak privilege control for agents. |
| ASI08 — Cascading Failures | Policy eviction can cascade across chained tasks and tool calls. | |
| Recommendation — Enforce per-action authorisation so agent privilege cannot depend on retained prompt text. Add durable checks at each handoff to stop control loss from propagating. | ||
| CSA MAESTRO | MAESTRO | MAESTRO addresses autonomy, orchestration and control boundaries for agentic systems. |
| Recommendation — Model policy as an external control in your agent threat model, not as memory content. | ||
| NIST AI RMF | Govern Map Measure Manage | AI RMF fits governance of high-impact AI behaviour when controls must persist across sessions. |
| Recommendation — Govern and measure agent behaviour with controls that survive context changes. | ||
| NIST SP 800-53 Rev 5 | AC-6 — Least Privilege | Context-only policy cannot reliably constrain privilege without durable access control. |
| Recommendation — Limit agent permissions to the minimum needed for the task and revoke unused access. | ||
Practitioner Guidance
What to prioritise: Treat any instruction that limits authority, approvals, data access, or external side effects as an enforcement requirement, not a prompt-writing exercise. If the action would be unacceptable after a context reset, it should not depend on context retention to stay safe.
What to verify: Check whether the agent can still be blocked, scoped, or forced to re-authorise after compaction, resumption, or multi-step delegation. The right test is simple: remove the remembered instruction and confirm that the control still holds.
Decision rule: If losing the instruction would change the permission boundary, move the rule into an external policy or approval path. If losing it would only reduce convenience, it may remain a prompt hint, but not a governance control.
Practitioner takeaway: A context-window policy is useful for guidance, but unsafe as the sole source of authority when the agent can act after its instructions have partially disappeared.