Adversarial Examples vs Prompt Injection: Why Defences, Testing and Ownership Differ

Security teams often file adversarial examples and prompt injection under one heading, "adversarial AI", and then apply the defences and test methods of one to the other, which leaves both exposed. An adversarial example is a numeric perturbation that pushes a predictive model across a decision boundary; prompt injection is natural-language instruction smuggled into a large language model (LLM) that then acts with the application's privileges. This guide does not re-explain either attack: it sets out why the defences, red-team methods and ownership differ, which controls transfer between the two, and how to organise testing so neither falls between teams. It is written for AI security leads, red team managers, ML engineers and application architects who run both classifiers and LLM agents.

Key takeaways

• Short answer: adversarial examples attack a model's decision, prompt injection attacks an application's authority. Evasion succeeds when a classifier returns the wrong label; injection succeeds when an LLM follows instructions it should have treated as data and then uses tools or data it was trusted with.
• The fixes sit in different layers. Adversarial training, input restoration and ensembles harden the model. For injection, the NCSC and OWASP both say full prevention is unlikely, so the main fix is architectural: privilege separation, tool restrictions and human approval.
• Some controls transfer, most do not. Query limits, telemetry and adaptive red teaming help against both. Adversarial training does little for indirect injection, and least-privilege tool design does nothing for a fooled malware classifier.
• Testing needs different skills and evidence. Evasion testing measures robust accuracy on a perturbed test set; injection testing measures whether untrusted content can trigger consequential actions, and the evidence is in tool-call logs, not model outputs.
• Ownership must be explicit. Model hardening usually belongs to the ML or data science team; injection resistance belongs to the application or platform architect and the identity team that scopes the agent's credentials. The red team tests both, against agreed scope.

Two attacks, two failure points

Both attacks are catalogued in the same frameworks, which is part of the confusion. NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, treats evasion of predictive models in section 2.2 and direct and indirect prompt injection against generative models in sections 3.3 and 3.4. MITRE ATLAS records evasion as AML.T0015 Evade AI Model, built with AML.T0043 Craft Adversarial Data, and injection as AML.T0051 LLM Prompt Injection with sub-techniques AML.T0051.000 Direct and AML.T0051.001 Indirect. MITRE ATT&CK v20 adds T1699 Direct Prompt Injection and T1698 Indirect Prompt Injection under Execution.

The useful distinction is where the system fails. With an adversarial example the model is the control: a phishing classifier, a malware detector or a face match. When it is wrong, the downstream decision is wrong, and nothing else needed to go wrong. With prompt injection the model is usually not a control at all; it is a component holding privileges. The failure is that a component which cannot reliably tell instructions from data was given authority to act. The UK NCSC frames this as a confused deputy problem and argues LLMs are inherently confusable, so the problem cannot be fixed the way SQL injection was fixed with parameterised queries.

The live Adversarial Examples and Evasion Attacks Guide covers how evasion works and where it shows up; the Prompt Injection Guide covers injection channels and payload hiding. This page sits between them.

Side-by-side comparison

Attacker knowledge and cost

For evasion, NIST AI 100-2 E2025 classifies attackers by knowledge of the model (white-box, gray-box, black-box) and ATLAS splits AML.T0043 into AML.T0043.000 White-Box Optimization, AML.T0043.001 Black-Box Optimization, AML.T0043.002 Black-Box Transfer and AML.T0043.003 Manual Modification. Optimisation usually needs many queries or a substitute model, which is why per-identity query limits and probing detection work for classifiers. Microsoft's paper on red teaming more than 100 generative AI products still lists "attackers can succeed without computing gradients" as a lesson: cheap manual changes often beat research-grade optimisation.

For prompt injection the knowledge bar is lower. The attacker needs to know only that a model will read their content and what tools it holds. That is why indirect injection scales: one poisoned web page or email can reach every agent that retrieves it, with no query budget to exhaust and no probing pattern for a rate limiter to catch.

The two meet in one place. Zou and colleagues (Universal and Transferable Adversarial Attacks on Aligned Language Models, 2023) used greedy and gradient-based search to build adversarial suffixes that made aligned LLMs comply with harmful requests, and showed suffixes optimised on open models transferred to black-box commercial chatbots. That is the adversarial example method applied to an LLM's safety behaviour, ATLAS AML.T0054 LLM Jailbreak. It matters for this comparison because it shows optimisation-style attacks are available to injection attackers too, so a filter tuned on hand-written injection phrases is not evidence of robustness. See Prompt Injection vs Jailbreaking for the injection versus jailbreak split.

Impact when models act through tools

A fooled classifier makes one wrong decision per input. A fooled agent may make a chain of decisions, each executed with real credentials. That difference changes the threat model more than any property of the attack itself.

• Evasion against a security model: the impact is bounded by what the model gates. A phishing filter that misses one message lets one message through; the user, mail authentication and endpoint controls still stand. ATLAS AML.T0015 notes the aim is often a traditional cyber objective such as getting malware past detection.
• Injection against a read-only chatbot: the impact is mostly content: misleading answers, leaked system prompts, brand damage. Serious, but bounded.
• Injection against an agent with tools: ATLAS AML.T0053 AI Agent Tool Invocation and AML.T0086 Exfiltration via AI Agent Tool Invocation describe the step from instruction to action. The EchoLeak 2025 Microsoft 365 Copilot case shows the pattern: a crafted email was pulled into context and the assistant placed internal data in a link the browser fetched automatically. Microsoft reported no evidence of exploitation and fixed it server-side, but the lesson is that the blast radius was set by what the assistant could reach.
• Evasion feeding an agent: the two can chain. An agent that relies on a classifier to label content as safe before acting inherits the classifier's evasion risk. Treat a classifier verdict as one signal, not a permission.

The identity angle is direct for injection. The agent's credentials, OAuth scopes and tool permissions define the worst case, so the identity team owns part of the fix. See the Tool-Call Security for AI Agents guide.

Defences that transfer and defences that do not

NIST AI 100-2 E2025 describes adversarial training as costly and says its benefit usually comes at the cost of lower accuracy on clean data, and that designing models that resist evasion while keeping accuracy remains an open problem. OWASP LLM01:2025 says it is unclear whether fool-proof prevention methods exist for prompt injection. Neither attack has a complete model-level fix, but the next-best controls are different.

The architectural answer for injection is set out in the paper Design Patterns for Securing LLM Agents against Prompt Injections (Beurer-Kellner and colleagues, 2025). Its core principle is that once an agent has ingested untrusted input, that input must not be able to trigger consequential actions. It proposes six patterns: Action-Selector, Plan-Then-Execute, LLM Map-Reduce, Dual LLM, Code-Then-Execute and Context-Minimization. Google DeepMind's CaMeL work (Defeating Prompt Injections by Design, 2025) takes the same line, separating control flow from data flow and using capabilities to enforce policy at tool calls; on the AgentDojo benchmark it reports solving 77% of tasks with provable security, against 84% for an undefended system. Nothing comparable exists for evasion, because a classifier's whole job is to read the untrusted input.

See the Secure Agent Design Patterns Guide for how to apply these patterns.

Testing approaches and red-team methods

Both disciplines share one lesson: defences must be evaluated against adaptive attackers. Carlini and colleagues (On Evaluating Adversarial Robustness, 2019) observed that most proposed defences against adversarial examples were quickly shown to be flawed. The 2025 paper The Attacker Moves Second repeated the finding for LLMs: using gradient descent, reinforcement learning, random search and human-guided exploration, the authors bypassed 12 recent jailbreak and prompt injection defences with attack success above 90% for most, where the defences had originally reported near-zero success. A defence tested only against a fixed attack list is not evidence.

Microsoft's red teaming paper adds two points that apply to both: know what the system can do and where it is deployed before testing, and red teaming is different from safety benchmarking. For agents, the first point is decisive: the test plan should start from the tool and permission inventory, not from a list of prompts. Scope, rules of engagement and data handling belong in a written plan; see the AI Red Team Test Plan Template and the AI Red Teaming Guide.

Who owns which risk

Ownership gaps are the most common practical failure. An ML team assumes the platform team handles LLM risks; the platform team assumes "adversarial AI" is the ML team's problem.

Record this split in the AI governance RACI. See the AI Governance RACI Template. Small organisations will often have one person in several columns; the point is that each row has a named owner, not that each column is a separate team.

Choosing where to invest

• You run classifiers that gate security or identity decisions and no agents: invest in evasion testing, query limits, output obfuscation and ensembles. Prompt injection matters only where an LLM sits in the pipeline.
• You run read-only LLM chat: injection and jailbreak testing focused on data leakage and content; guardrails and provider alignment carry more weight because there are no tools to restrict.
• You run agents with tools or connectors: injection is the priority. Spend first on privilege separation, restricting tools on untrusted data and human approval for consequential actions. Guardrails are a supplementary layer.
• You run agents that call classifiers: test the chain. An evaded classifier label should never be the only thing standing between untrusted content and a privileged action.
• Large organisation: separate evasion and injection test tracks with shared reporting. Small organisation: one AI security owner, one test plan with two sections, and external testing for the model track if no ML specialist is in-house.

Worked scenario: a bank's onboarding pipeline

A bank runs a document and face-match model for customer onboarding, and an LLM agent that reads applicant emails, summarises them and can update the case record and send templated replies.

1. Threat model: the face-match model faces evasion (AML.T0015) from fraudsters altering images; the agent faces indirect injection (AML.T0051.001) from text hidden in applicant emails and attachments.
2. Model track: the ML team adds an adversarial evaluation set, returns only pass, fail or refer to the case system, rate-limits submissions per applicant and device, and keeps a liveness check and human review as second signals.
3. Agent track: the architect applies Plan-Then-Execute: the agent's actions are fixed from the case worker's request before it reads the email. The agent's identity can update only the summary field and send only approved templates; status changes require case-worker approval. Outbound links and images in replies are blocked.
4. Testing: the red team runs two tracks: perturbed images against the model, and planted instructions in emails and PDFs against the agent, scoring whether any reached a status change or data disclosure.
5. Result: neither track's defences would have helped the other. Adversarial training does nothing for the email path; the scoped agent identity does nothing for an altered photo.

Detection and measurement

• Evasion signals: high query volume per caller, near-duplicate inputs, scores clustered just over the acceptance threshold, disagreement between ensemble members.
• Injection signals: the NCSC recommends logging inputs, outputs and tool or API calls, and notes failed calls may reveal an attacker refining an approach. Watch for tool calls unrelated to the user's request, outbound requests to new domains and reads of data the task did not need.

Metrics:

• Robust accuracy against an adaptive test set, per model release, beside clean accuracy.
• Share of agents whose consequential tools require approval or are blocked once untrusted content enters context (target 100%).
• Injection scenarios that reached a consequential action in the latest test cycle (target zero).
• Share of model endpoints and agents with a named owner in the RACI for each risk.
• Time from a confirmed evasion or injection finding to fix in production.

Framework mapping

Common failure patterns

• Treating injection as an input-filtering problem: buying a prompt classifier and calling the agent protected. The adaptive-attack results above show filters are bypassed; the tool permissions decide the damage.
• Using a jailbreak benchmark as the injection test: a model that refuses harmful content can still follow a polite planted instruction to forward a file.
• Assuming model secrecy stops evasion: transfer attacks from substitute models work without access to the target.
• Testing defences with yesterday's attack list: both research lines show static evaluations overstate robustness.
• No owner for the agent's credentials: the ML team hardens the model while the agent keeps broad delegated scopes nobody reviews. See Prompt injection data theft for how this plays out.
• Letting a classifier verdict authorise an agent action: chaining two fallible models without a deterministic check.

Practitioner checklist

• Inventory every predictive model that gates a decision and every LLM application or agent, with tools and credentials.
• Assign an owner for evasion risk and an owner for injection risk on each, in the AI governance RACI.
• For classifiers: rate-limit and attribute every caller, return labels not scores externally, and keep a second signal for high-impact decisions.
• For agents: apply a design pattern that stops untrusted input triggering consequential actions, and scope the agent identity to the task.
• Require approval or block tool use once untrusted content enters context, for every consequential tool.
• Run two red-team tracks with distinct pass criteria, against adaptive attackers, on every model or tool change.
• Log model inputs and scores for classifiers, and prompts, retrieved content and tool calls for agents, into the SOC.
• Never let a model verdict alone authorise a privileged action.

Standards and references

• NIST: AI 100-2 E2025 Adversarial Machine Learning, A Taxonomy and Terminology of Attacks and Mitigations
• MITRE: ATLAS (AML.T0015, AML.T0043, AML.T0051 and mitigations)
• OWASP: LLM01:2025 Prompt Injection
• NCSC: Prompt injection is not SQL injection (it may be worse)
• arXiv: Design Patterns for Securing LLM Agents against Prompt Injections
• arXiv: Defeating Prompt Injections by Design (CaMeL)
• arXiv: The Attacker Moves Second, Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections
• arXiv: On Evaluating Adversarial Robustness

Related NHI Mgmt Group resources: Adversarial Examples and Evasion Attacks Guide · Prompt Injection Guide · Multimodal Prompt Injection Guide · Prompt Injection Incident Response Playbook · AI Red Teaming Guide