An input that has been deliberately altered so a machine learning model misclassifies it while it still appears normal to a human reviewer. The change may be tiny, but it is designed to exploit the model’s sensitivity to specific feature patterns rather than to break the surrounding system.
Expanded Definition
An adversarial example is a deliberately perturbed input crafted to cause a machine learning model to produce the wrong output while the input still looks ordinary to a human. The attack targets the model’s decision boundary, not the underlying infrastructure, so it can succeed even when endpoint, network, and application controls are intact. In practice, this term is used most often in computer vision, speech, and text systems, though usage in the industry is still evolving for multimodal and agentic AI workflows. NIST’s AI Risk Management Framework and related guidance treat this kind of behaviour as a risk to model robustness, safety, and trustworthiness, while threat catalogues such as the MITRE ATLAS adversarial AI threat matrix describe how attackers manipulate model inputs to evade detection or force misclassification.
The key distinction is intent: a noisy or corrupted input is not necessarily adversarial unless it is engineered to exploit model weaknesses. The most common misapplication is calling any model error an adversarial example, which occurs when a benign edge case, data-quality issue, or distribution shift is mistaken for a targeted attack.
Examples and Use Cases
Implementing defences against adversarial examples rigorously often introduces latency, extra testing overhead, and higher false-positive rates, requiring organisations to weigh model resilience against operational simplicity.
- Image classification in security screening, where slight pixel-level changes can cause a model to label a restricted object as benign.
- Fraud or identity workflows, where subtle input manipulation can steer an automated decisioning model away from a high-risk classification, intersecting with verification controls described in the NIST SP 800-63 Digital Identity Guidelines.
- Prompted AI assistants, where crafted text can push a downstream classifier or guardrail model into an unsafe or incorrect response path, a pattern increasingly discussed in adversarial ai analysis.
- Content moderation pipelines, where a minor alteration to symbols, spacing, or wording can bypass automated detection while remaining legible to a reviewer.
- Operational red teaming, where defenders use the MITRE ATLAS adversarial AI threat matrix to design tests that reveal brittle model behaviour before production exposure.
Why It Matters for Security Teams
Adversarial examples matter because they expose a gap between human perception and machine decision-making. A model can appear accurate in standard validation and still fail under targeted manipulation, which means security teams cannot rely on headline accuracy alone. The operational risk is highest where AI outputs influence access decisions, fraud screening, moderation, malware triage, or incident prioritisation. In those settings, a single manipulated input can become a control bypass, not just a model mistake.
This is where AI security and identity governance intersect. If an AI system helps decide who or what is trusted, then adversarial examples can distort NHI onboarding, identity verification, or access approval paths. Defenders should combine robust training, adversarial testing, logging, and compensating controls such as human review for high-impact cases, alongside control mapping to NIST SP 800-53 Rev 5 Security and Privacy Controls and monitoring guidance from CISA cyber threat advisories. Organisations typically encounter the full impact only after a manipulated input has already altered a decision, at which point adversarial example resilience becomes operationally unavoidable to address.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATLAS address the attack and risk surface, while NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | Addresses AI robustness and harmful model behavior, which includes adversarial example risk. | |
| NIST AI 600-1 | GenAI profile guidance covers trustworthy AI behavior and adversarial resilience concerns. | |
| MITRE ATLAS | Catalogues adversarial AI techniques, including input manipulation and evasion methods. |
Assess model robustness and add testing, monitoring, and governance for adversarial manipulation.
Related resources from NHI Mgmt Group
- How should security teams test AI models for adversarial manipulation?
- Why do traditional IAM controls fall short for adversarial ML risk?
- What is the difference between prompt injection testing and model adversarial testing?
- When do adversarial prompts become a business risk rather than a model-quality issue?
