A tactic in which untrusted content changes an AI agent’s behaviour by altering the instructions it follows. This can occur through prompt injection, malicious tool output, or hidden directives inside web pages and documents, turning normal workflow inputs into control inputs.
Expanded Definition
Instruction manipulation describes a class of attacks where content that should be treated as data is used to influence an AI agent’s execution path. The risk emerges when the system does not reliably separate user intent, system instructions, tool output, and retrieved content. In agentic workflows, that boundary matters because an agent may follow hidden directives embedded in a webpage, document, email, or tool response as if they were trusted operator commands.
Definitions vary across vendors and research communities, but the security meaning is consistent: the attacker is not trying to break the model itself, but to steer its behaviour by corrupting the instruction hierarchy. This makes the term especially relevant to AI assistants that can browse, call tools, write files, or trigger downstream actions. Guidance in the NIST Cybersecurity Framework 2.0 supports this boundary-focused view by emphasising governance, access control, and risk management around system interactions.
The most common misapplication is treating all model misuse as prompt injection, which occurs when organisations ignore tool output, retrieval content, or copied instructions that can also alter agent behaviour.
Examples and Use Cases
Implementing protections against instruction manipulation rigorously often introduces workflow friction, requiring organisations to weigh agent autonomy and productivity against stronger content inspection and policy enforcement.
- A support bot reads a customer email containing hidden text that instructs it to reveal internal policy details, so the email must be handled as untrusted input, not as instruction.
- An AI research assistant retrieves a webpage that embeds malicious directives inside HTML comments or formatted text, causing it to ignore prior system rules unless content is sanitised.
- A code-generation agent receives tool output from a repository scanner that includes adversarial instructions, showing why tool responses need the same trust scrutiny as user prompts.
- A document-processing workflow ingests a PDF with concealed commands that redirect the agent to send data to an external endpoint, making output filtering and action gating essential.
- A knowledge-assistant uses retrieval-augmented generation and later follows a poisoned passage over the system policy, demonstrating why retrieval sources need provenance and filtering controls aligned with OWASP guidance for LLM applications.
Why It Matters for Security Teams
Instruction manipulation matters because it breaks the assumption that an AI agent will execute only the organisation’s approved instructions. Once an attacker can steer the agent through content, they may alter outputs, exfiltrate sensitive data, trigger unsafe tool calls, or bypass approval steps. That turns a productivity feature into an execution-risk problem.
Security teams need to design for trust boundaries, provenance checks, policy separation, and least-privilege tool access. This is where AI governance connects directly with identity and access management: an agent with broad tool permissions can do far more damage if manipulated than a constrained one. Defensive patterns commonly discussed in the OWASP Top 10 for LLM Applications and NIST AI risk guidance focus on limiting what untrusted content can influence, especially when agents can act on behalf of users or services. For operational resilience, teams also need logging, review gates, and clear escalation paths when a model behaves as if it has received new instructions from an untrusted source.
Organisations typically encounter the blast radius only after an agent has already taken the wrong action or exposed data, at which point instruction manipulation becomes operationally unavoidable to address.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST AI RMF, NIST AI 600-1 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | Covers agentic AI attack paths including prompt and instruction injection. | |
| NIST AI RMF | Defines AI risk governance practices applicable to instruction manipulation. | |
| NIST AI 600-1 | GenAI profile addresses risks from prompt and content injection into model workflows. | |
| NIST CSF 2.0 | PR.AC-4 | Access control supports limiting what manipulated agents can reach or change. |
| OWASP Non-Human Identity Top 10 | Non-human identities and agent credentials amplify the impact of instruction manipulation. |
Harden agent workflows so untrusted content cannot override system or developer instructions.
Related resources from NHI Mgmt Group
- What breaks when agent frameworks and instruction files are not lifecycle-governed?
- How can teams reduce the impact of instruction smuggling in LLM pipelines?
- Who is accountable when an AI assistant performs a sensitive action after DOM manipulation?
- How do security teams know when an AI instruction file has become a security control?
