Model Context Poisoning

What Model Context Poisoning Means

Model context poisoning is not just “bad input.” It is a runtime trust failure where an AI system accepts manipulated context as if it were reliable, then reasons, summarizes, or acts on it. The poisoned material can enter through prompts, retrieval results, memory, tool responses, or other context channels the model treats as authoritative.

The danger is that the model may remain internally consistent while being externally steered. Because the compromise happens in the working context rather than the model weights, the attack can be subtle, transient, and difficult to distinguish from ordinary user input or normal retrieval noise.

In practice, this makes context poisoning a security problem about provenance, trust boundaries, and runtime control, not simply about prompt quality. It also explains why the same issue can affect chat assistants, retrieval-augmented systems, and agentic workflows that consume tool output or shared memory.

How Poisoned Context Changes Model Behaviour

Poisoned context can bias the model toward incorrect facts, unsafe recommendations, hidden instructions, or attacker-favoured actions. If the system blends retrieved documents, memory, and live tool output without strong separation, the model may treat a malicious fragment as one more legitimate source and amplify it across later turns.

The attack works best when the system gives context broad authority. A poisoned memory item can survive across sessions, a manipulated retrieval result can outrank better evidence, and a compromised tool response can override the user’s intent. In agentic systems, that can turn a false premise into an incorrect decision or an unsafe tool call.

• AI Agent Memory Poisoning Guide is useful where poisoned memory persists across sessions or projects.
• OWASP Agentic Top 10 ASI06 Guide helps explain how memory and context poisoning affects agent design, provenance, and rollback.

Where Context Poisoning Enters the Stack

Common entry points include prompt injection hidden in user content, malicious documents added to retrieval corpora, poisoned vector-store entries, manipulated memory, and untrusted tool output. The issue is broader than a single prompt because the model may combine multiple sources and infer a coherent answer from them even when one source is compromised.

Systems that support long-lived memory or cross-session reuse are especially exposed. So are deployments that accept content from third-party connectors, shared workspaces, or external tools without clear write controls and provenance checks. The more the application treats context as reusable knowledge, the more attractive it becomes as an attack surface.

AI Agent Memory Security Guide covers isolation, write controls, logging, and retention for context sources that should not be blindly trusted.

Detection, Recovery, and Hardening

Detection usually starts with provenance review: identify which context source introduced the suspicious instruction, assertion, or behavioural drift. Teams should compare the model’s output against trusted ground truth, inspect retrieval logs and memory writes, and look for sudden style changes, policy bypasses, or repeated references to low-quality sources.

Recovery often means removing or quarantining the poisoned context, rolling back memory or retrieval indexes, and validating that the system returns to trusted behaviour. Hardening focuses on limiting who can write to memory, separating untrusted content from trusted instructions, logging context ingestion, and treating tool output as data until it is validated. AI Model Poisoning Incident Playbook is helpful when compromise is suspected and rollback or retraining decisions are needed.

OWASP LLM Top 10 LLM04 Data and Model Poisoning Guide is useful when the same poisoning pattern reaches training data, RAG corpora, or model intake.

Risk and Threat Considerations

Model context poisoning is dangerous because it attacks the model’s decision-making inputs rather than the model itself. That can lead to misinformation, policy bypass, unsafe automation, data leakage, or attacker-controlled actions that look like ordinary model behaviour.

Failure mechanism: The attacker slips false or malicious context into a channel the system trusts, then relies on the model to prioritise that material during retrieval, reasoning, or action selection.

Impact: The model may produce deceptive outputs, follow attacker-authored instructions, or persist the compromise across later interactions if the poisoned context is stored or reused.

Practitioner Guidance

Why practitioners should care: Treat context sources as security-sensitive inputs, not passive helper data. If the system can read from memory, retrieval stores, documents, or tools, then those channels need explicit trust rules and monitoring.

What to watch for: Focus on unexpected instruction patterns, unusual retrieval ranking, repeated policy overrides, and context that changes behaviour without a corresponding user request. If the system can be influenced by shared or persistent context, assume poisoning is a live design risk rather than a rare edge case.

AI Supply Chain Security and AI-BOM Guide is helpful for tracking which models, datasets, tools, and connectors can introduce tainted context into the system.

Related resources from NHI Mgmt Group

• Why do context poisoning attacks matter if the model itself is secure?
• What is the Model Context Protocol (MCP) and why does it matter for security?
• How should security teams govern AI agents that use Model Context Protocol?
• Why does Model Context Protocol create identity risk for enterprises?