AI lowers the cost of believable deception and makes appearance less reliable as a signal of identity. That means organisations should rely less on what a request looks or sounds like and more on controlled verification steps, scoped approvals, and device or workflow-based confirmation.
Why This Matters for Security Teams
AI-driven deception changes the trust model behind verification. Voice clones, synthetic video, generated chat, and prompt-driven social engineering can all make a request look legitimate while bypassing the human cues that staff traditionally use to judge intent. That shifts the problem from spotting a fake person to proving that a request is authorised, contextual, and traceable. Current guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls supports stronger authentication, approval integrity, and monitoring, but organisations still need to adapt those controls to AI-enabled deception paths.
This matters across fraud, helpdesk recovery, payment approval, executive impersonation, and agentic ai workflows. If a policy still treats voice, tone, writing style, or a familiar chat interface as a strong signal, it is already behind the threat. Verification now needs to answer three questions: who is acting, from what device or workflow, and under what approval state. In practice, many security teams encounter these failures only after a synthetic request has already triggered payment, access reset, or data exposure, rather than through intentional verification design.
How It Works in Practice
Organisations should treat verification as a layered control, not a single challenge-response step. AI attacks are effective because they compress the time needed to create convincing pretexts and can adapt the wording, tone, and channel to match the target. That means verification must rely on evidence that is harder to fake at scale: authenticated sessions, managed devices, scoped entitlements, workflow approvals, and post-action logging.
A practical pattern is to separate identity proofing from action approval. A user may be known, but the specific request still needs contextual validation. For example, a request to change payment details should not rely on a voice call alone; it should require a separate approval path, a verified device, and a callback or in-app confirmation that is bound to the original ticket. For AI agents, the same logic applies with even more force because the agent may act autonomously across tools and APIs. The most useful question is not whether the request sounds plausible, but whether the actor is authorised to perform that action in that workflow.
- Use strong authentication for entry, then require step-up verification for sensitive actions.
- Bind approvals to a ticket, workflow, or signed request rather than free-form chat or email.
- Limit the blast radius with least privilege and short-lived access.
- Monitor for anomalous approval paths, identity changes, and rapid sequence abuse.
Adversary behavior is well documented in the MITRE ATT&CK Enterprise Matrix, while AI-specific attack paths are increasingly mapped in the MITRE ATLAS adversarial AI threat matrix and reinforced by real incident reporting such as Anthropic — first AI-orchestrated cyber espionage campaign report. These controls tend to break down when legacy helpdesk processes, informal approvals, and unmanaged communication channels are allowed to bypass the formal verification workflow because attackers simply choose the weakest channel.
Common Variations and Edge Cases
Tighter verification often increases friction, so organisations have to balance user experience against the cost of a failed trust decision. Best practice is evolving, and there is no universal standard for every workflow yet, especially where automation, customer support, and executive operations overlap. The right answer depends on whether the action is reversible, financially sensitive, or capable of granting further access.
One common edge case is recovery and escalation. If a user loses access, the fallback path can become the easiest path for an attacker using AI-generated pretexts. Another is high-volume operations, where teams are tempted to relax checks so work keeps moving. That shortcut usually increases risk, especially when approvals happen over email, chat, or ad hoc calls. For agentic AI, the issue is broader: a model may produce a correct-looking request, but that does not mean it is entitled to execute it. Organisations should define which actions require human approval, which can be pre-authorised, and which must never be automated.
Where personal or financial identity is involved, the verification standard should be even stricter because AI attacks can combine synthetic media with stolen data to pass shallow checks. The safer pattern is to prefer workflow-bound confirmation, device trust, and logged approvals over confidence in the message itself. In low-risk contexts, lighter controls may be acceptable, but the threshold should be deliberate, documented, and reviewed.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST CSF 2.0, NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.AA-01 | Verification must prove actor legitimacy before sensitive actions proceed. |
| NIST AI RMF | AI deception creates model and workflow trust risks that need governance. | |
| OWASP Agentic AI Top 10 | Agentic systems can be manipulated into executing unauthorised actions. | |
| MITRE ATLAS | Adversarial AI tactics explain how synthetic content and prompts deceive controls. | |
| NIST AI 600-1 | GenAI-specific misuse changes how organisations should validate outputs and requests. |
Map AI deception paths to adversarial techniques and add targeted detection.
Related resources from NHI Mgmt Group
- Why do AI agents change the way organisations think about zero trust?
- Why do AI-driven attacks change the way security teams should think about containment?
- Why do AI-assisted attacks change the way IAM teams think about approvals?
- Why do AI-enabled attackers change the way organisations should think about access control?
