Machine learning models that decide whether a file is malware, an email is phishing, a transaction is fraud or a face matches an ID can be fooled by inputs crafted to be misclassified, and those models increasingly sit in front of identity and security decisions. These crafted inputs are called adversarial examples, and using them to slip past a model is an evasion attack. This guide explains the attack at concept level, where it appears in security and identity systems, how to detect probing and evasion, and which hardening measures hold up according to NIST and MITRE ATLAS. It is written for security engineers, ML engineers and owners of fraud, malware and identity verification models. It deliberately contains no attack recipes. For the wider AI threat picture see the AI Threats and Model Security Guide, and for the full NIST taxonomy the Adversarial ML Taxonomy Guide.
Key takeaways
- Evasion happens at inference time. The attacker does not touch training data; they alter the input so a deployed model gives the wrong answer, often with changes a human would not notice.
- Black-box access is enough. NIST AI 100-2 E2025 describes query-only attacks as the most practical setting, and substitute models let attackers transfer attacks they built offline.
- No defence is complete. NIST lists adversarial training, randomized smoothing and formal verification as the more resilient classes, all with accuracy and cost trade-offs, and says robust, accurate models remain an open problem.
- Design the system, not just the model. Query limits, reduced output detail, ensembles, multiple signals and a human or rule-based backstop do more in production than any single robustness technique.
- Probing is detectable. Attackers usually need many queries to find a working input. Per-identity query telemetry on model APIs is the most useful early signal.
How the attack works
NIST AI 100-2 E2025 (section 2.2) defines an evasion attack as one in which the adversary generates adversarial examples: inputs whose classification can be changed to a class of the attacker's choosing, usually with a small perturbation. MITRE ATLAS records the same behaviour as AML.T0015 Evade AI Model, which uses AML.T0043 Craft Adversarial Data, and notes the goal is often a traditional cyber attack such as getting malware past AI-based detection.
What the attacker knows about the model shapes the attack, and therefore the defence.
| Attacker knowledge (NIST AI 100-2 section 2.1.4) | What they have | ATLAS technique | What limits it |
|---|---|---|---|
| White-box | Architecture, parameters, often training data | AML.T0043.000 White-Box Optimization | Keeping model artefacts private (AML.M0001), access control on model files (AML.M0005) |
| Gray-box | Partial knowledge, for example features or architecture but not weights | AML.T0043 Craft Adversarial Data | Limiting published technical detail, feature secrecy |
| Black-box, query based | Only an inference API; scores or labels returned | AML.T0043.001 Black-Box Optimization | Query limits (AML.M0004), less detailed outputs (AML.M0002), detection (AML.M0015) |
| Black-box, transfer | A substitute model they built and fully control | AML.T0043.002 Black-Box Transfer, AML.T0005 Create Proxy AI Model | Ensembles (AML.M0006), robustness training (AML.M0003) |
- Score versus label leakage: NIST distinguishes score-based black-box attacks, which use confidence scores, from decision-based attacks, which see only the final label. Returning a raw score makes the attacker's search easier.
- Transferability: NIST section 2.2.3 describes attackers training a substitute model, crafting examples against it and transferring them to the target. Secrecy of your model alone is therefore not a defence.
- Verify then deploy: ATLAS AML.T0042 Verify Attack notes an adversary can verify an attack once, offline or through an API, then use it against many copies of the model, for example on edge devices.
- Physical world: where models read sensors or cameras, ATLAS AML.T0041 Physical Environment Access covers changes made in the real world, and NIST cites printed objects such as glasses frames and road sign stickers as studied examples.
Where it shows up
| Model type | What evasion achieves | Documented example (MITRE ATLAS case study) |
|---|---|---|
| ML malware detection | Malicious file classified as benign | AML.CS0003 Bypassing Cylance's AI Malware Detection (research exercise, 2019); AML.CS0014 Confusing Antimalware Neural Networks (Kaspersky research, gray-box) |
| Email and phishing classifiers | Phishing email or page scored as safe | AML.CS0008 ProofPoint Evasion (CVE-2019-20634, research using a copy-cat model); AML.CS0032 adversarial examples seen in logs of a commercial phishing webpage detector (incident, 2022) |
| Network and DNS detection | Command and control traffic or generated domains missed | AML.CS0000 and AML.CS0001 (DGA detection evasion research) |
| Identity verification and biometrics | Fraudulent identity accepted | AML.CS0017 Bypassing ID.me Identity Verification (incident: at least 180 false unemployment claims, at least $3.4 million paid) |
| Fraud and risk scoring | Fraudulent transaction or sign-up approved | Same pattern: attackers adjust controllable features until the score passes |
| Vision models at the edge | Misclassification of images or objects | AML.CS0011 Microsoft Edge AI Evasion (Azure Red Team exercise) |
NIST notes that domain constraints make attacks on cybersecurity data harder than on images, because a modified file must still run and a modified email must still deliver its message. That constraint helps defenders, but the case studies show it does not prevent evasion.
Impact
- Silent control failure: the detector reports nothing, so the downstream attack (malware execution, credential phishing, fraudulent onboarding) proceeds with no alert.
- Identity assurance failure: an evaded face match or document check creates a real account for a false identity, which then receives credentials and access. See the Deepfake-Resistant Identity Verification Guide.
- Scaled abuse: once one working input or perturbation pattern is found, it can often be reused at scale, as ATLAS AML.T0042 describes.
- Erosion of trust in automation: teams lose confidence in model verdicts and fall back on manual review, increasing cost.
Detection
Evasion leaves traces before it succeeds: the search for a working input needs many queries, and successful inputs are often statistically unusual. ATLAS AML.M0015 Predictive AI Adversarial Input Detection recommends detecting and blocking adversarial inputs or atypical queries that deviate from benign behaviour, and AML.M0024 AI Telemetry Logging recommends logging model inputs and outputs.
| Signal | Where to collect it | Why it matters |
|---|---|---|
| High query volume per caller identity, API key or account | Model API gateway logs | Black-box optimisation needs many queries |
| Many near-duplicate inputs with small differences | Input hashing or embedding similarity on the request log | Classic probing pattern around a decision boundary |
| Inputs scored just over the acceptance threshold, clustered | Model output logs | Successful evasion often lands just inside "allow" |
| Disagreement between models or signals | Ensemble outputs, secondary rules, sandbox verdicts | An input that fools one model often fails others |
| Rising downstream misses | Incident data: confirmed malware, phishing or fraud later found that the model passed | Measures real-world evasion rate |
| Input distribution drift | Feature statistics over time | Can show a campaign probing or exploiting the model |
Metrics:
- Robust accuracy on an adversarial test set, tracked per model release, alongside clean accuracy.
- Share of model API callers with query rate limits and identity attribution (target 100%).
- Number of probing alerts per month and the share investigated.
- Rate of confirmed bad items that the model scored as benign (false negatives found later).
- Time from a confirmed evasion to retrained or patched model in production.
Prevention and hardening
Model level
- Adversarial training (ATLAS AML.M0003 Predictive AI Model Hardening): NIST describes adding adversarial examples with correct labels during training. It is costly and usually reduces accuracy on clean data, so measure both.
- Randomized smoothing and formal verification: NIST lists both as giving certified guarantees, but smoothing certifies only some inputs and verification is limited by scale and compute. Use them for small, high-stakes models.
- Input restoration (AML.M0010): preprocessing that removes or reverses perturbations. Test it against adaptive attackers, not just known attacks.
- Evaluate against adaptive attacks: NIST notes many proposed defences were later broken and says mitigations need to be evaluated against strong adaptive attacks. A defence tested only on yesterday's attack is not evidence.
System level
- Restrict queries (AML.M0004): rate and volume limits per caller, with every model API caller authenticated as a distinct identity, human or machine. See the Rate Limiting and API Abuse Guide.
- Reduce output detail (AML.M0002): return a decision or coarse band, not a raw score, to external callers.
- Ensembles and multiple signals (AML.M0006, AML.M0009): combine models of different families and non-ML signals (signatures, sandboxing, reputation, liveness checks), so one evasion is not enough.
- Limit release of model detail (AML.M0001): do not publish architectures, thresholds or checkpoints used in production.
- Backstops: keep human review or rule-based checks for high-impact decisions such as account opening and high-value payments.
Rollout
- Inventory every model that makes or supports a security, fraud or identity decision, with its owner and exposure (internal, partner, public API).
- Add caller identity, logging and query limits to each model API first; these are cheap and reversible.
- Build an adversarial evaluation set with your red team or a testing tool, under an agreed scope. See the AI Red Teaming Guide.
- Pilot robustness training on one model, compare clean and robust accuracy, and roll back to the previous model if business accuracy drops below the agreed threshold.
- Add ensembles or secondary signals for the highest-impact decisions.
- Repeat evaluation on every model release and after any confirmed evasion.
Framework mapping
| Need | Reference |
|---|---|
| Threat model and terminology | NIST AI 100-2 E2025 section 2.2; MITRE ATLAS AML.T0015, AML.T0043 |
| Testing before release | NIST SP 800-53 SA-11 Developer Testing and Evaluation, CA-8 Penetration Testing |
| Input handling | SP 800-53 SI-10 Information Input Validation; ATLAS AML.M0010, AML.M0015 |
| Monitoring for probing | SP 800-53 SI-4 System Monitoring; CSF 2.0 DE.AE-02; ATLAS AML.M0024 |
| Risk assessment of model weaknesses | SP 800-53 RA-3; CSF 2.0 ID.RA-01 |
Lessons from incidents
NIST section 2.2.4 cites real-world evasion including fraud against an identity verification service, with more than 80,000 attempts in late 2020, and a phishing detector evaded with simple cropping, masking or blurring rather than optimisation. MITRE ATLAS records the identity verification case as AML.CS0017: an individual filed at least 180 false unemployment claims by getting stolen identities through automated photo-to-ID matching, and received at least $3.4 million. Two lessons follow:
- Cheap evasion is common: real attackers often use simple transformations, not research-grade optimisation. Test your models against these first.
- Identity decisions need layered evidence: a single automated match should not be the only gate for creating an identity that will receive money or access. See the Identity Proofing and KYC Guide.
No breach in the NHI Mgmt Group database has yet been confirmed as a classic adversarial example attack, so none is linked here.
Practitioner checklist
- List every model that makes security, fraud or identity decisions and record who can query it.
- Authenticate every model API caller as a distinct identity and apply per-caller query limits.
- Return decisions or coarse bands externally, not raw confidence scores.
- Log inputs, outputs and caller identity, and alert on probing patterns.
- Maintain an adversarial evaluation set and run it on every release, including simple transformations.
- Track clean and robust accuracy side by side when using adversarial training.
- Combine model verdicts with independent signals for high-impact decisions.
- Keep production model details, thresholds and checkpoints out of public material.
Standards and references
- NIST: AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
- MITRE ATLAS: Evade AI Model
- MITRE ATLAS: Craft Adversarial Data
- MITRE ATLAS: Predictive AI Adversarial Input Detection
- MITRE ATLAS: Predictive AI Model Hardening
- MITRE ATLAS: Bypassing ID.me Identity Verification
- NIST: SP 800-53 Rev. 5
Related NHI Mgmt Group resources: AI Threats and Model Security Guide · Adversarial ML Taxonomy Guide · AI Data Poisoning and Backdoors Guide · AI Red Teaming Guide