The failure mode is context laundering. When summarisation preserves hostile instructions as if they were normal user feedback, the agent can later act on them without recognising the manipulation. That breaks review-based governance because the malicious input has already been compressed into legitimate-looking state before a human can inspect it.
How context compaction fails when hostile text is treated as state
Context compaction is supposed to preserve what matters and discard noise, but the failure is not just loss of detail. When summarisation rewrites attacker input into apparently normal memory, the system is no longer preserving evidence of manipulation. It is converting untrusted content into durable operational context, which changes how later decisions are made.
That matters because agentic systems often reuse compacted context as if it were an approved operating brief. If the original prompt injection, misleading instruction, or social-engineering content is flattened into a neutral summary, the agent can inherit the attacker’s intent without any visible marker that the source was malicious.
This is why the issue is best understood as a trust boundary problem inside the agent lifecycle. The compaction step is not merely compression, it is a governance act that decides which statements become authoritative for future tool use, delegation, and policy interpretation.
Why compaction can launder attacker intent
In practice, compaction usually favors relevance, continuity, and brevity. Those goals are useful, but they can also erase provenance. A hostile instruction can survive if it is paraphrased into a businesslike summary, especially when the model is trained to preserve “helpful” task direction rather than source attribution.
The dangerous pattern is that the summary looks like legitimate user state, not adversarial input. Once that happens, downstream reasoning may treat the attacker’s text as settled context instead of an object that still needs review, challenge, or quarantine.
The problem becomes sharper in multi-step workflows, where the compacted record is reused across turns, tools, or sub-agents. One compromised interpretation can persist long after the original prompt is out of view, which makes later review too late to prevent the policy break.
For practitioners, the key design question is whether the compaction layer preserves enough traceability to distinguish user intent, system instruction, and externally supplied text. If it does not, summarisation can become an unwitting trust amplifier. See NHIMG’s AI Agents vs Agentic AI for the autonomy boundary that makes this failure mode more consequential, and Agentic AI Security Guide for the broader control surfaces that compaction can undermine.
How to spot the break before it reaches tools
The visible symptom is not always obvious malice. More often, the red flag is a compacted state object that contains instructions, preferences, or “resolved” decisions with no clear source trail. If the summary cannot show what came from the user, what came from the model, and what was inferred, governance has already been weakened.
Another warning sign is when the agent begins following a compressed instruction that was never explicitly re-authorised after summarisation. That is especially risky when the instruction changes tool scope, approval thresholds, or the meaning of prior user requests.
Compaction should therefore be treated as a security-relevant transformation, not a formatting convenience. Once hostile content is indistinguishable from legitimate memory, the system loses the ability to challenge it at the point where challenge still matters.
Operationally, this is where auditability and memory hygiene intersect. A compacted context should remain inspectable enough to explain why the agent later acted. If it cannot, the system is relying on a hidden decision path that humans cannot meaningfully review.
Risk and Threat Considerations
Context laundering creates a quiet privilege problem: the attacker does not need to win at the moment of injection if the compaction process later promotes the malicious text into trusted state. That can defeat human review, policy checks, and post-hoc investigation because the evidence of manipulation has been normalised before oversight can intervene.
Failure mechanism: Summarisation strips source distinction and rephrases hostile instructions as benign context, so later agent decisions consume attacker-authored intent as if it were approved memory.
Impact: The agent may repeat, amplify, or operationalise the attacker’s instruction across tools, sessions, or delegated actions, creating durable compromise and making detection materially harder.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST CSF 2.0 sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI06 — Memory & Context Poisoning | Compaction can preserve hostile text as trusted context. |
| ASI09 — Human-Agent Trust Exploitation | The failure exploits human trust in seemingly normal agent state. | |
| Recommendation — Isolate and validate memory so poisoned context cannot steer later actions. Require explicit review boundaries before agent state is treated as trusted. | ||
| NIST CSF 2.0 | PR.AA-05 — Identities Are Proofed, Bound, and Reauthenticated | Compacted context needs re-authentication of trust before reuse in later actions. |
| Recommendation — Re-establish trust boundaries before reusing compressed agent state. | ||
Practitioner Guidance
What to prioritise: Preserve provenance across compaction. A compacted state is only safe if it still distinguishes quoted user content, model inference, and system policy so reviewers can see what was asserted versus what was trusted.
What to verify: Check whether summaries can be traced back to source turns and whether any instruction-bearing text is re-authorised after compression. If the answer is no, treat the compaction layer as a control gap, not a harmless optimisation.
What not to automate: Do not allow a summariser to silently upgrade untrusted text into policy-bearing memory. The decision to promote a statement into durable agent state needs human-reviewed rules or an explicit trust boundary, not just better paraphrasing.
Practitioner takeaway: The control objective is not perfect compression, it is preserving the ability to tell trusted memory from attacker influence before the agent acts on it.