Policy Puppetry is an attack that uses crafted prompt text to make a model follow an attacker’s fake policies instead of the real ones. It targets systems that rely on weak instruction hierarchy or poorly separated context. Defenders check for instruction injection, constrain tool use, and keep system policies isolated from untrusted text.
How Policy Puppetry Works
Policy puppetry is a prompt injection pattern that tries to make the model obey attacker-written rules as if they were higher priority than the system’s real instructions. The attack succeeds when the application blurs the boundary between trusted policy and untrusted text, especially in chat logs, retrieved content, or mixed-context prompts.
The core weakness is not the wording itself, but the instruction hierarchy around it. If the model cannot reliably separate policy from content, an attacker can smuggle fake governance into the conversation and cause the model to follow it.
Why It Is Dangerous
This attack can redirect model behavior without needing to break the underlying system directly. In practice, that may lead to unsafe outputs, bypassed refusals, leaked system instructions, or tool actions that were never meant to be available to the user’s input.
It is especially dangerous in systems that let model output drive business actions, support workflows, or downstream automation. The model may appear compliant while actually following attacker-authored policy fragments embedded in the prompt stream.
Where It Shows Up
Policy puppetry often appears in applications that assemble prompts from multiple sources, such as user messages, retrieved documents, tickets, emails, or agent memory. The attack surface grows when the application treats those sources as equally trustworthy or passes them into the model with weak role separation.
It is closely related to instruction injection, but the emphasis is on counterfeit policy framing rather than simple malicious commands. The attacker is trying to impersonate governance, not just ask for an unsafe answer.
How Defenders Reduce the Attack Surface
Defensive design should keep system policy isolated from untrusted text and make the trust boundary explicit. Strong prompt structure, constrained tool invocation, and clear separation between instructions, data, and retrieved context all reduce the chance that attacker text can masquerade as policy.
Review prompt assembly paths for places where user-controlled content can shape control instructions, refusal logic, or tool-routing decisions. Validation should focus on whether the model can be induced to reinterpret authority, not just whether obvious jailbreak phrases are blocked.
Risk and Threat Considerations
Policy puppetry creates a control-plane risk for AI applications, because the attacker is targeting the rules the model uses to decide what to trust. Once fake policy text is accepted as authoritative, the compromise can affect safety behavior, tool access, and any downstream action that depends on model judgment.
Failure mechanism: The model fails to preserve instruction hierarchy and treats attacker-controlled text as if it were a governing policy statement, especially when the prompt mixes instructions with retrieved or user-supplied content.
Impact: The attacker can suppress safeguards, steer tool use, alter responses, or induce data exposure and workflow abuse without changing the application’s code or backend permissions.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATLAS and OWASP Agentic AI Top 10 address the attack and risk surface, while NIST AI RMF sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | Govern | AI risk governance must separate trusted policy from untrusted prompt content. |
| Recommendation — Define prompt trust boundaries and review AI controls for instruction hierarchy failures. | ||
| MITRE ATLAS | Prompt Injection | ATLAS catalogs prompt-injection techniques that steer model behavior through crafted text. |
| Recommendation — Map suspicious prompt patterns to prompt-injection techniques and test for policy override attempts. | ||
| OWASP Agentic AI Top 10 | ASI01 — Agent Goal Hijack | Goal hijacking covers adversarial steering of agent behavior away from intended objectives. |
| ASI02 — Tool Misuse | Policy puppetry can manipulate tool decisions by altering what the agent believes it is allowed to do. | |
| ASI09 — Human-Agent Trust Exploitation | The attack exploits trust in human-like policy language to influence agent decisions. | |
| Recommendation — Harden agent prompts so untrusted text cannot redirect the agent’s governing objective. Constrain tool calls so prompt content cannot expand tool authority. Separate user-authored text from governance logic to prevent trust abuse. | ||
Practitioner Guidance
Why practitioners should care: Policy puppetry is a prompt-assembly problem as much as a model-behavior problem. If the application does not separate policy, context, and user input cleanly, the model can be manipulated into treating the wrong source as authoritative.
What to watch for: Look for prompts that blend governance text with retrieved content, agent instructions, or user-authored rules. Those designs deserve extra testing for instruction hierarchy failures and tool-routing abuse.
Practitioner takeaway: Treat policy text as protected control material, not just another prompt fragment, and verify that the model cannot be persuaded to promote untrusted content into an instruction source.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Free weekly newsletter
Subscribe to the NHI & AI Identity Journal
The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.
Bonus 33% off our NHI Course when you subscribe.
Reviewed and updated by the NHIMG editorial team on October 17, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org