Join our Newsletter — 33% off our NHI Course
Home› Guides› Adversarial Examples and Evasion Attacks Guide: Detect and…
Guide Agent Threats & Attacks

Adversarial Examples and Evasion Attacks Guide: Detect and Harden ML Models Against Evasion

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 27 October 2026 10 min read
On this page

Machine learning models that decide whether a file is malware, an email is phishing, a transaction is fraud or a face matches an ID can be fooled by inputs crafted to be misclassified, and those models increasingly sit in front of identity and security decisions. These crafted inputs are called adversarial examples, and using them to slip past a model is an evasion attack. This guide explains the attack at concept level, where it appears in security and identity systems, how to detect probing and evasion, and which hardening measures hold up according to NIST and MITRE ATLAS. It is written for security engineers, ML engineers and owners of fraud, malware and identity verification models. It deliberately contains no attack recipes. For the wider AI threat picture see the AI Threats and Model Security Guide, and for the full NIST taxonomy the Adversarial ML Taxonomy Guide.

Key takeaways

  • Evasion happens at inference time. The attacker does not touch training data; they alter the input so a deployed model gives the wrong answer, often with changes a human would not notice.
  • Black-box access is enough. NIST AI 100-2 E2025 describes query-only attacks as the most practical setting, and substitute models let attackers transfer attacks they built offline.
  • No defence is complete. NIST lists adversarial training, randomized smoothing and formal verification as the more resilient classes, all with accuracy and cost trade-offs, and says robust, accurate models remain an open problem.
  • Design the system, not just the model. Query limits, reduced output detail, ensembles, multiple signals and a human or rule-based backstop do more in production than any single robustness technique.
  • Probing is detectable. Attackers usually need many queries to find a working input. Per-identity query telemetry on model APIs is the most useful early signal.

How the attack works

NIST AI 100-2 E2025 (section 2.2) defines an evasion attack as one in which the adversary generates adversarial examples: inputs whose classification can be changed to a class of the attacker's choosing, usually with a small perturbation. MITRE ATLAS records the same behaviour as AML.T0015 Evade AI Model, which uses AML.T0043 Craft Adversarial Data, and notes the goal is often a traditional cyber attack such as getting malware past AI-based detection.

What the attacker knows about the model shapes the attack, and therefore the defence.

Attacker knowledge (NIST AI 100-2 section 2.1.4)What they haveATLAS techniqueWhat limits it
White-boxArchitecture, parameters, often training dataAML.T0043.000 White-Box OptimizationKeeping model artefacts private (AML.M0001), access control on model files (AML.M0005)
Gray-boxPartial knowledge, for example features or architecture but not weightsAML.T0043 Craft Adversarial DataLimiting published technical detail, feature secrecy
Black-box, query basedOnly an inference API; scores or labels returnedAML.T0043.001 Black-Box OptimizationQuery limits (AML.M0004), less detailed outputs (AML.M0002), detection (AML.M0015)
Black-box, transferA substitute model they built and fully controlAML.T0043.002 Black-Box Transfer, AML.T0005 Create Proxy AI ModelEnsembles (AML.M0006), robustness training (AML.M0003)
  • Score versus label leakage: NIST distinguishes score-based black-box attacks, which use confidence scores, from decision-based attacks, which see only the final label. Returning a raw score makes the attacker's search easier.
  • Transferability: NIST section 2.2.3 describes attackers training a substitute model, crafting examples against it and transferring them to the target. Secrecy of your model alone is therefore not a defence.
  • Verify then deploy: ATLAS AML.T0042 Verify Attack notes an adversary can verify an attack once, offline or through an API, then use it against many copies of the model, for example on edge devices.
  • Physical world: where models read sensors or cameras, ATLAS AML.T0041 Physical Environment Access covers changes made in the real world, and NIST cites printed objects such as glasses frames and road sign stickers as studied examples.

Where it shows up

Model typeWhat evasion achievesDocumented example (MITRE ATLAS case study)
ML malware detectionMalicious file classified as benignAML.CS0003 Bypassing Cylance's AI Malware Detection (research exercise, 2019); AML.CS0014 Confusing Antimalware Neural Networks (Kaspersky research, gray-box)
Email and phishing classifiersPhishing email or page scored as safeAML.CS0008 ProofPoint Evasion (CVE-2019-20634, research using a copy-cat model); AML.CS0032 adversarial examples seen in logs of a commercial phishing webpage detector (incident, 2022)
Network and DNS detectionCommand and control traffic or generated domains missedAML.CS0000 and AML.CS0001 (DGA detection evasion research)
Identity verification and biometricsFraudulent identity acceptedAML.CS0017 Bypassing ID.me Identity Verification (incident: at least 180 false unemployment claims, at least $3.4 million paid)
Fraud and risk scoringFraudulent transaction or sign-up approvedSame pattern: attackers adjust controllable features until the score passes
Vision models at the edgeMisclassification of images or objectsAML.CS0011 Microsoft Edge AI Evasion (Azure Red Team exercise)

NIST notes that domain constraints make attacks on cybersecurity data harder than on images, because a modified file must still run and a modified email must still deliver its message. That constraint helps defenders, but the case studies show it does not prevent evasion.

Impact

  • Silent control failure: the detector reports nothing, so the downstream attack (malware execution, credential phishing, fraudulent onboarding) proceeds with no alert.
  • Identity assurance failure: an evaded face match or document check creates a real account for a false identity, which then receives credentials and access. See the Deepfake-Resistant Identity Verification Guide.
  • Scaled abuse: once one working input or perturbation pattern is found, it can often be reused at scale, as ATLAS AML.T0042 describes.
  • Erosion of trust in automation: teams lose confidence in model verdicts and fall back on manual review, increasing cost.

Detection

Evasion leaves traces before it succeeds: the search for a working input needs many queries, and successful inputs are often statistically unusual. ATLAS AML.M0015 Predictive AI Adversarial Input Detection recommends detecting and blocking adversarial inputs or atypical queries that deviate from benign behaviour, and AML.M0024 AI Telemetry Logging recommends logging model inputs and outputs.

SignalWhere to collect itWhy it matters
High query volume per caller identity, API key or accountModel API gateway logsBlack-box optimisation needs many queries
Many near-duplicate inputs with small differencesInput hashing or embedding similarity on the request logClassic probing pattern around a decision boundary
Inputs scored just over the acceptance threshold, clusteredModel output logsSuccessful evasion often lands just inside "allow"
Disagreement between models or signalsEnsemble outputs, secondary rules, sandbox verdictsAn input that fools one model often fails others
Rising downstream missesIncident data: confirmed malware, phishing or fraud later found that the model passedMeasures real-world evasion rate
Input distribution driftFeature statistics over timeCan show a campaign probing or exploiting the model

Metrics:

  • Robust accuracy on an adversarial test set, tracked per model release, alongside clean accuracy.
  • Share of model API callers with query rate limits and identity attribution (target 100%).
  • Number of probing alerts per month and the share investigated.
  • Rate of confirmed bad items that the model scored as benign (false negatives found later).
  • Time from a confirmed evasion to retrained or patched model in production.

Prevention and hardening

Model level

  • Adversarial training (ATLAS AML.M0003 Predictive AI Model Hardening): NIST describes adding adversarial examples with correct labels during training. It is costly and usually reduces accuracy on clean data, so measure both.
  • Randomized smoothing and formal verification: NIST lists both as giving certified guarantees, but smoothing certifies only some inputs and verification is limited by scale and compute. Use them for small, high-stakes models.
  • Input restoration (AML.M0010): preprocessing that removes or reverses perturbations. Test it against adaptive attackers, not just known attacks.
  • Evaluate against adaptive attacks: NIST notes many proposed defences were later broken and says mitigations need to be evaluated against strong adaptive attacks. A defence tested only on yesterday's attack is not evidence.

System level

  • Restrict queries (AML.M0004): rate and volume limits per caller, with every model API caller authenticated as a distinct identity, human or machine. See the Rate Limiting and API Abuse Guide.
  • Reduce output detail (AML.M0002): return a decision or coarse band, not a raw score, to external callers.
  • Ensembles and multiple signals (AML.M0006, AML.M0009): combine models of different families and non-ML signals (signatures, sandboxing, reputation, liveness checks), so one evasion is not enough.
  • Limit release of model detail (AML.M0001): do not publish architectures, thresholds or checkpoints used in production.
  • Backstops: keep human review or rule-based checks for high-impact decisions such as account opening and high-value payments.

Rollout

  1. Inventory every model that makes or supports a security, fraud or identity decision, with its owner and exposure (internal, partner, public API).
  2. Add caller identity, logging and query limits to each model API first; these are cheap and reversible.
  3. Build an adversarial evaluation set with your red team or a testing tool, under an agreed scope. See the AI Red Teaming Guide.
  4. Pilot robustness training on one model, compare clean and robust accuracy, and roll back to the previous model if business accuracy drops below the agreed threshold.
  5. Add ensembles or secondary signals for the highest-impact decisions.
  6. Repeat evaluation on every model release and after any confirmed evasion.

Framework mapping

NeedReference
Threat model and terminologyNIST AI 100-2 E2025 section 2.2; MITRE ATLAS AML.T0015, AML.T0043
Testing before releaseNIST SP 800-53 SA-11 Developer Testing and Evaluation, CA-8 Penetration Testing
Input handlingSP 800-53 SI-10 Information Input Validation; ATLAS AML.M0010, AML.M0015
Monitoring for probingSP 800-53 SI-4 System Monitoring; CSF 2.0 DE.AE-02; ATLAS AML.M0024
Risk assessment of model weaknessesSP 800-53 RA-3; CSF 2.0 ID.RA-01

Lessons from incidents

NIST section 2.2.4 cites real-world evasion including fraud against an identity verification service, with more than 80,000 attempts in late 2020, and a phishing detector evaded with simple cropping, masking or blurring rather than optimisation. MITRE ATLAS records the identity verification case as AML.CS0017: an individual filed at least 180 false unemployment claims by getting stolen identities through automated photo-to-ID matching, and received at least $3.4 million. Two lessons follow:

  • Cheap evasion is common: real attackers often use simple transformations, not research-grade optimisation. Test your models against these first.
  • Identity decisions need layered evidence: a single automated match should not be the only gate for creating an identity that will receive money or access. See the Identity Proofing and KYC Guide.

No breach in the NHI Mgmt Group database has yet been confirmed as a classic adversarial example attack, so none is linked here.

Practitioner checklist

  • List every model that makes security, fraud or identity decisions and record who can query it.
  • Authenticate every model API caller as a distinct identity and apply per-caller query limits.
  • Return decisions or coarse bands externally, not raw confidence scores.
  • Log inputs, outputs and caller identity, and alert on probing patterns.
  • Maintain an adversarial evaluation set and run it on every release, including simple transformations.
  • Track clean and robust accuracy side by side when using adversarial training.
  • Combine model verdicts with independent signals for high-impact decisions.
  • Keep production model details, thresholds and checkpoints out of public material.

Standards and references

Related NHI Mgmt Group resources: AI Threats and Model Security Guide · Adversarial ML Taxonomy Guide · AI Data Poisoning and Backdoors Guide · AI Red Teaming Guide

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 27 October 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.