Governance decay in AI agents shows where policy enforcement belongs

At a glance

What this is: This is an analysis of governance decay in long-horizon AI agents, where context compaction can silently drop in-context safety and policy instructions.

Why it matters: It matters because IAM, PAM, and AI governance teams cannot rely on prompts or memory alone to preserve high-consequence controls across long-running agent sessions.

Context

Governance decay is the failure mode that appears when an AI agent’s operating rules live inside the same context window that gets compressed for continuity. The problem is not that the model suddenly changes character. The problem is that the enforcement signal is treated like expendable history, so the control can disappear while the task continues.

For identity and access programmes, the key question is where policy should reside when the actor is an agent rather than a human. In-context instructions are useful for steering, but they are not a reliable control plane for compliance boundaries, tool scope, or data egress rules. That makes this topic directly relevant to agentic AI governance and to any IAM or PAM team deciding where enforcement should actually live.

Key questions

Q: What breaks when agent policies live only in the context window?

A: The control breaks when compaction or eviction removes the rule that was supposed to constrain the agent. The model may still be the same, but the behaviour changes because the instruction is no longer present in the retained state. For high-consequence actions, that means prompt-based governance is too fragile to rely on as the only enforcement layer.

Q: Why does governance decay create a compliance risk for AI agents?

A: Because compliance rules, data boundaries, and approval requirements can disappear during routine summarisation even when the task continues normally. A policy that can be dropped by the harness is not durable enough for regulated or sensitive workflows. That makes the risk operational, not theoretical, especially in long-running sessions that cross many turns.

Q: How can security teams tell whether an agent is failing because of compaction?

A: Look for behaviour changes that line up with transcript compression, summarisation thresholds, or memory eviction rather than with a new prompt, model version, or user request. The signal is often a sudden return to ungoverned defaults after the conversation grows long. That is a harness issue, not proof of model drift.

Q: Should agent governance be enforced in prompts or at the gateway boundary?

A: Use the gateway boundary for controls that cannot be allowed to vanish, such as outbound data restrictions, tool permissions, and budget limits. Keep prompts for steering, not for enforcing the controls that must survive compaction. The clean split is durable enforcement outside the context and conversational guidance inside it.

Technical breakdown

How context compaction erases standing agent rules

Context compaction is a summarisation pass that rewrites long conversation history into a smaller, task-preserving form. That works for continuity, but it is hostile to policy because the summariser is optimising for the current objective, not for the preservation of older constraints. If a rule says never send a contract externally, the compactor may decide the task state matters more than the prohibition and drop the prohibition from the retained summary. Once that happens, the agent has not been bypassed in the traditional sense. It is simply operating without the rule that previously governed it.

Practical implication: Treat any policy that must survive long sessions as an out-of-band control, not as text that can be summarised away.

Why governance decay is a harness problem, not a model problem

The article’s central technical point is that the loss happens in the surrounding system, not in the base model weights. That distinction matters because teams often over-attribute these failures to model quality, prompt design, or benchmark performance. Governance decay can affect stronger models too, because the weak point is the lifecycle of the policy text inside the agent harness. This is why standard model evaluation can miss it: the benchmark may test what the model knows, while the deployment failure comes from what the harness later removes.

Practical implication: Shift assurance testing from model behaviour alone to harness behaviour, especially around retention of safety-critical instructions.

Why enforcement belongs at the request path, not only in the prompt

The architectural answer is to move consequential controls to a layer every call already crosses. A gateway, policy engine, or other trusted enforcement boundary can evaluate the action at request time rather than trusting the prompt to remember itself. That allows data boundaries, tool scope, spend limits, and other durable constraints to be applied consistently even after compaction. In parallel, in-context rules still have a role, but mainly for steering tone, workflow, or task framing. The dividing line is whether the policy can tolerate being forgotten.

Practical implication: Place high-consequence controls on the request path and reserve in-context instructions for low-risk steering.

Threat narrative

Attacker objective: The attacker wants the agent to carry out a prohibited action by making the governing rule disappear from the retained context.

1. Entry occurs when an adversary places content in the agent’s context, giving them an opportunity to influence what the compaction step preserves.
2. Credential or policy abuse follows when compaction drops the protective instruction and the agent continues with ungoverned defaults.
3. Impact occurs when the agent performs an action that the original rule was meant to prevent, such as sending sensitive material outside the organisation.

Breaches seen in the wild

• LiteLLM MCP auth bypass 2026: An exploited LiteLLM MCP auth bypass and default sk-1234 master keys let attackers steal AI gateway master and provider API keys.
• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Governance decay is an enforcement placement problem, not a prompting problem. Once a control can be summarised away, it is not a durable control. That is true whether the rule governs data egress, tool scope, or spending. For agentic AI programmes, the decisive question is not how elegantly the policy is written, but where it lives and whether compaction can reach it.

The failure mode creates a new governance concept: context-window fragility. In long-horizon systems, any rule that exists only inside the retained conversation inherits the same volatility as the transcript itself. This makes agent governance qualitatively different from human IAM, where the policy store is external by design. Practitioners should treat compacted context as an unreliable control surface and classify policies by whether they survive summarisation.

This is where NHI and agentic AI governance converge. An AI agent can be a software actor whose permissions and task scope must be governed like a non-human identity, but its behavioural instructions are still mediated through a transient memory layer. That means identity, privilege, and policy enforcement must be separated. The durable part belongs in externally enforced access control, while the transient part can remain in-context only for steering.

Compaction-aware design is becoming a prerequisite for trustworthy AI operations. If the control model assumes that all important instructions remain visible, it will fail under normal long-session operation. The research also shows that adversarially placed content can accelerate the loss of constraints, which turns a maintenance feature into an attack surface. Practitioners should design for forgotten rules, not merely for injected ones.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Authorisation Guide

What this signals

Context-window fragility: enterprises are moving into a phase where the security question is no longer whether an agent can follow instructions, but whether those instructions survive the system’s own housekeeping. That shifts governance from prompt engineering to control placement, which is a more durable way to think about compliance boundaries and blast-radius reduction.

Where agent behaviour depends on retention inside a shrinking window, the programme risk is that policy and memory compete for the same token budget. The practical response is to move decision rights, authorisation checks, and data-boundary enforcement to infrastructure that does not participate in compaction, while leaving the conversation layer to handle only steering and task continuity.

For practitioners

• Move high-consequence policies out of context Put data egress limits, tool scope, approval thresholds, and spend rules on an external enforcement path that compaction cannot rewrite.
• Classify in-context rules as steering only Reserve prompts and memory for tone, task framing, and low-risk preferences, then document which controls are not acceptable to keep only in the transcript.
• Instrument compaction as a governed event Log when history is summarised, what policy text survives, and when behaviour changes around a compaction boundary so investigators can distinguish forgetting from injection.
• Scope agent privileges by identity and task Tie each agent or tool session to an explicit identity, least-privilege scope, and task boundary so the request path can enforce limits even after context loss.

Key takeaways

• Governance decay is a control-placement problem, not just a prompt-design problem, because compaction can silently remove the very rules that agents were following.
• The article’s evidence shows that a routine harness function can turn into a policy failure surface when long-horizon behaviour depends on retained context.
• The strongest mitigation is to enforce high-consequence controls outside the context window and to keep in-context instructions limited to steering.

Key terms

• Governance decay boundary: The point at which the controls meant to constrain an agent become editable by that same agent or its improvement loop. Once that happens, policy text and runtime behaviour can drift together. The remedy is to move enforcement outside the editable surface.
• Context Compaction: Context compaction is the process of summarising or pruning conversation history so a long-running AI session can continue inside a fixed token window. It is useful for continuity, but it can also remove standing instructions, making it a security-sensitive operation in agentic systems.
• Out-of-Band Enforcement: Out-of-band enforcement is control that occurs outside the model prompt or memory, such as at a gateway, policy engine, or request boundary. It matters because controls in this layer are not subject to summarisation, eviction, or prompt manipulation in the same way as in-context rules.
• Context-Window Fragility: Context-window fragility is the tendency of in-context controls to weaken or disappear as a conversation grows and the system compresses history. It is a useful way to describe why controls that appear stable in short tests may fail during ordinary long-horizon operation.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through the NHI Foundation Level course, the industry's only accredited NHI security programme. It is designed for practitioners who need to govern machine identities, access scope, and policy enforcement across modern AI and identity programmes.