When context or memory is mutable and untrusted, the agent’s future decisions can be steered by attacker-controlled inputs. That breaks the assumption that an agent’s next action is based on authorised intent. Teams then lose reliable control over what the agent believes, which tool it chooses, and what data it may expose.
Why This Matters for Security Teams
When an agent’s context or memory can be rewritten, the security problem is no longer limited to access control. The attacker is influencing the agent’s next decision, which means prompt injection, poisoned memory, and tampered retrieval can redirect tool use, alter outputs, or expose data the agent was never meant to reach. This is a core agentic risk, not a side effect.
That is why guidance from the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework focuses on runtime control, data provenance, and trustworthy orchestration rather than static trust in model output. NHIMG’s AI Agents: The New Attack Surface report shows how quickly agent behaviour escapes intended scope when controls are weak. In practice, many security teams encounter this only after the agent has already acted on poisoned context, rather than through intentional testing.
How It Works in Practice
Manipulated memory usually breaks systems in three places: retrieval, reasoning, and action. If an attacker can seed a vector store, poison conversation history, or alter persisted summaries, the agent may treat hostile instructions as trusted state. If the agent then uses that state to plan, it may select the wrong tool, broaden scope, or expose secrets during an otherwise normal workflow.
Security teams need to separate untrusted inputs from durable agent state. Best practice is evolving, but current guidance suggests treating memory as a controlled asset with provenance, integrity checks, and expiration rules. Where possible, use short-lived context, scoped retrieval, and explicit trust boundaries between user content, system instructions, and persistent memory. Runtime policy enforcement matters more than pre-approved “safe” prompts because the dangerous decision is often made after retrieval, not at prompt entry.
Operational controls that help include:
- Sign or otherwise verify high-value memory objects before reuse.
- Store only the minimum durable context needed for the task.
- Apply policy checks at retrieval time, not just at write time.
- Log which memory items influenced which tool calls and outputs.
- Revise secrets handling so credentials are never recoverable from memory or summaries.
NHIMG research on OWASP NHI Top 10 aligns with this problem because agent memory corruption often becomes an identity and authorization failure after the fact. The CSA MAESTRO agentic AI threat modeling framework and MITRE ATLAS adversarial AI threat matrix both reinforce that adversarial manipulation of model inputs and state must be assumed in design. These controls tend to break down when memory is shared across tenants or reused across long-running workflows because poisoned state can persist beyond the original attack window.
Common Variations and Edge Cases
Tighter memory controls often increase latency and operational overhead, requiring organisations to balance persistence against auditability and speed. That tradeoff matters because some teams need durable memory for customer support, code generation, or multi-step automation, while others can safely keep only short-lived session context.
There is no universal standard for how much memory should be trusted yet. In high-risk workflows, best practice is to treat persistent memory as untrusted until validated, especially when retrieval can cross projects, users, or sensitivity domains. Systems that blend chat history, tool output, and external knowledge are especially vulnerable because a single poisoned item can influence many later decisions.
Edge cases include delegated agents, multi-agent swarms, and retrieval-augmented pipelines where one agent writes state that another agent later consumes. Those architectures increase blast radius because compromise is no longer limited to one conversation. NHIMG’s AI LLM hijack breach coverage and the Anthropic report on AI-orchestrated cyber espionage both show why attackers exploit the weakest state boundary, not just the model itself. For this reason, organisations should assume that any mutable memory can become an attack surface unless it is explicitly governed, versioned, and bounded by policy.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and CSA MAESTRO address the attack and risk surface, while NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A3 | Covers prompt injection and poisoned context that steer agent decisions. |
| CSA MAESTRO | TH-03 | Addresses adversarial manipulation of agent state and memory. |
| NIST AI RMF | GOVERN | Governance is required when memory can alter autonomous behaviour. |
Validate retrieved context and isolate untrusted inputs before they influence agent actions.
