Context manipulation is a tactic in which an attacker alters the conversational or tool-use context so the model treats malicious instructions as legitimate. It includes patterns such as conversation spoofing and tool call spoofing. Defenders use this category to group similar attacks and design broader mitigations.
Expanded Definition
Context manipulation is a prompt- and orchestration-layer attack pattern where the attacker changes what the model believes is true about the current session, tool state, or message provenance. In practice, that can mean injecting fake prior messages, spoofing a trusted tool output, or replaying instructions so the model treats hostile content as authorised.
The term is broader than a single jailbreak technique. It covers conversation spoofing, tool call spoofing, and other forms of contextual deception that exploit how the model weighs recency, role labels, and apparent system history. The key boundary is that the attacker is not merely asking for a bad answer; they are reshaping the model’s operational context. Guidance is still evolving on how best to classify these patterns, but the security consensus is clear that provenance and trust boundaries matter as much as the instruction text itself.
For readers working with agentic systems, this distinction is critical because the model may act on context as if it were verified state. The most common misunderstanding is to treat these attacks as pure content filtering problems, when the real issue is trust in message origin and tool output authenticity.
Examples and Use Cases
Context manipulation shows up whenever a model consumes multi-turn dialogue, external tools, or retrieved content and must decide what to trust.
- A malicious user inserts fabricated “earlier” instructions into a conversation so the assistant follows those instead of the visible prompt.
- An attacker makes a tool response look like it came from a trusted internal service, causing the model to accept false status or policy data.
- In a retrieval-augmented workflow, untrusted content is formatted to resemble system guidance or operator notes, altering the model’s interpretation of the task.
- In an agent workflow, a spoofed tool call result can steer the agent into unsafe next steps, including sending data to the wrong destination or approving an action prematurely.
The tradeoff is that richer context improves usefulness, but every additional context source expands the trust surface. Teams that centralise orchestration gain consistency, but they also create stronger incentives for attackers to target the context layer itself.
Security Implications
When context manipulation succeeds, the model can be induced to ignore legitimate constraints, reveal sensitive data, or take actions that appear internally justified. The failure condition is often not a weak model response in isolation, but a broken chain of trust between user input, system instructions, retrieved content, and tool outputs.
That can produce downstream effects such as policy bypass, data leakage, unsafe tool invocation, or escalation from a harmless conversation into an authorised workflow action. In agentic environments, the blast radius is larger because the model may not just answer incorrectly; it may issue commands, modify records, or relay compromised instructions into other systems.
A practical signal is inconsistent provenance handling: if the system cannot reliably distinguish a user message from a tool message or a retrieved passage from an operator directive, the model is being asked to infer trust from formatting instead of verification.
Domain and Governance Relevance
Context manipulation matters most in AI security and agentic workflow governance, where the central question is not only what the model can say, but what it can be persuaded to believe as authorised. That makes message provenance, tool authenticity, and context separation core design concerns rather than optional hardening.
For identity and access governance, the relevance becomes sharper when non-human identities, API tokens, or delegated tool permissions are part of the execution path. A spoofed context can cause an agent to misuse legitimate machine credentials or to treat an untrusted instruction as if it were part of a trusted operating chain. In NHI-heavy environments, the trust model for context is therefore inseparable from the trust model for credentials and tool access.
From an NHIMG perspective, the operational lesson is that context should be treated as a governed asset with provenance rules, not as free-form text. Where tool use is involved, the question is always who asserted the instruction, through which channel, and under what authenticated state.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATLAS, OWASP Agentic AI Top 10 and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST AI RMF and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| MITRE ATLAS | AML.TA0001 — Input Manipulation | Context manipulation alters model inputs and trusted context. |
| Recommendation — Map spoofed context patterns to AML.TA0001 and detect input tampering in your AI pipeline. | ||
| OWASP Agentic AI Top 10 | A2 — Tool Misuse | Spoofed tool outputs can steer agent actions through false context. |
| Recommendation — Apply A2 controls to verify tool responses before an agent acts on them. | ||
| OWASP Non-Human Identity Top 10 | NHI-05 — Secrets and Credential Abuse | Spoofed context can trick agents into misusing machine credentials or tokens. |
| Recommendation — Bind NHI-05 governance to tool identities and prevent context from authorising secret use. | ||
| NIST AI RMF | MAP — Measure, Analyze, and Manage | Context manipulation is a governance issue in AI risk management. |
| Recommendation — Use MAP to classify context trust boundaries and manage prompt and tool provenance risk. | ||
| CIS Controls v8 | 6 — Access Control Management | Context spoofing can lead to unauthorised actions through trusted workflows. |
| Recommendation — Revoke or constrain workflow access so forged context cannot trigger privileged actions. | ||
