Deadly Duo

What the deadly duo means in practice

The deadly duo is dangerous because it pairs two ingredients that should rarely be combined: untrusted input and privileged action. The model is exposed to attacker-controlled content, then uses that content to make decisions or take actions through tools, databases, workflows, or other systems. The risk is not just bad text generation, but destructive behaviour executed with the system’s authority.

This pattern matters most in LLM applications that let model outputs influence operations. If the model can read external prompts, retrieved documents, tickets, emails, or web content and then act on them, the untrusted material can shape downstream behaviour even when no private data is directly exposed.

Why untrusted input plus action is so hazardous

The core problem is trust collapse. A model may treat malicious instructions as if they were legitimate user intent, especially when those instructions are embedded in content the system was designed to process. That can lead to destructive queries, unwanted tool calls, workflow abuse, or changes to records and settings that the attacker never touched directly.

The hazard increases when the model’s action surface is broad. Even a well-contained model can become unsafe if it can submit requests, approve operations, modify objects, or route decisions into business systems. The issue is therefore architectural: the model becomes a decision point, and the decision point is being fed adversarial material.

Common failure modes and attack paths

Deadly duo incidents usually follow a simple chain. The attacker supplies crafted content, the model ingests it, the content changes the model’s reasoning or plan, and the model performs an action that benefits the attacker. The action may be direct, such as sending a malicious request, or indirect, such as triggering a downstream workflow that causes damage later.

Typical failure modes include prompt injection, tool misuse, and privilege abuse through delegation. The model does not need access to sensitive data for the pattern to be dangerous, because the harm can come from issuing harmful commands, altering state, deleting records, or initiating side effects in connected systems.

This is why agentic AI controls increasingly focus on OWASP Agentic AI Top 10 and on NIST AI Risk Management Framework guidance for governing autonomy, tool use, and downstream impact.

Controls that reduce exposure

Reducing deadly duo risk starts with separating reading from acting. Systems should treat untrusted content as data, not instructions, and should constrain what the model is allowed to do after processing that content. The smaller the action surface, the harder it is for hostile input to become a harmful operation.

Practitioners also need strong authorization boundaries around tools, workflows, and service calls. The model should only be able to invoke actions that are explicitly intended, observable, and bounded by policy. Strong control design is especially important where the model touches operational systems, because NIST SP 800-53 Rev 5 Security and Privacy Controls and NIST Cybersecurity Framework 2.0 both emphasise bounded access, monitoring, and resilience.

How to detect and investigate misuse

Detection should focus on the action trail, not only the prompt text. If an LLM system suddenly begins issuing unusual tool calls, destructive queries, unexpected approvals, or commands that do not fit the normal task flow, those are strong indicators that untrusted input may have influenced behaviour.

Post-incident review should ask two questions: what content did the model ingest, and what authority did the model have when it acted? That review often reveals that the compromise was less about model failure alone and more about excessive trust placed in model-driven operations. For threat mapping and investigation depth, MITRE ATT&CK Enterprise Matrix remains useful for understanding attacker objectives and follow-on activity.

Risk and Threat Considerations

Deadly duo is a material security risk because it turns adversarial content into a possible trigger for privileged behaviour. The danger is not limited to data leakage, it includes destructive side effects, workflow abuse, and actions that occur with the system’s own authority.

Failure mechanism: The model consumes attacker-controlled input, misinterprets it as instruction or context, and then performs a harmful action through connected tools or workflows.

Impact: The result can be destructive queries, unauthorized changes, malicious automation, or broader compromise of business processes that rely on the model’s decisions.

Practitioner Guidance

Why practitioners should care: The deadly duo is a design problem, not just a prompt problem. If a system can both read untrusted content and take privileged action, security teams should treat that combination as a high-risk control boundary.

Governance implication: Owners should define which inputs are trusted, which actions the model may take, and where human approval or policy enforcement is required before side effects are allowed. That accountability should sit with the system that grants action authority, not only with the model vendor or prompt designer.

Practitioner takeaway: The safest pattern is to narrow authority, validate intent before execution, and assume that any content the model reads may be adversarial unless proven otherwise.

Related resources from NHI Mgmt Group

• How should security teams evaluate Duo Security alternatives for IAM governance?
• Why do Duo OTPs and similar one-time codes still fail against phishing?
• How do you detect Cisco Duo MFA bypass code use?
• Duo Security