TL;DR: AI-generated deepfakes and fabricated content are making social engineering and fraud more persuasive by exploiting emotion, bias, and misplaced trust in what people see and hear, according to Anomali. The control problem is shifting from spotting fakes to building resilience against manipulation before users act on it.
At a glance
What this is: This is Anomali’s analysis of how AI-generated disinformation and deepfakes strengthen social engineering by exploiting emotion, bias, and trust in familiar faces, voices, and messages.
Why it matters: It matters because identity, fraud, and security teams need controls that reduce the chance of human acceptance of forged content, not just tools that flag it after the fact.
By the numbers:
- One analysis reported a tenfold rise in deepfake related fraud between 2022 and 2023, as cited by Anomali.
Context
AI-generated disinformation is a governance problem because it targets the human decision layer, not just the technical control layer. When fabricated audio, video, or text can trigger trust, urgency, or fear, standard awareness training and single-channel verification become weaker than the attack path itself. In identity and fraud programmes, this overlaps with impersonation risk, out-of-band verification, and the reliability of evidence used to approve requests.
Anomali’s article frames deepfakes as a modern social engineering enabler rather than a standalone media problem. That distinction matters for practitioners because the same manipulation pattern can support payment fraud, executive impersonation, election influence, blackmail, and account compromise, and the starting point is often ordinary user trust rather than a technical exploit.
Key questions
Q: What breaks when people rely on deepfake detection as the main control?
A: Human spotting fails because even careful observers can miss convincing synthetic audio, video, or text. A detection-first model also assumes the fake is noticed before action, which is too late in urgent social engineering. Organisations need provenance checks, callback procedures, and approval controls that do not depend on instinct alone.
Q: Why do deepfakes make social engineering more effective?
A: Deepfakes make social engineering more effective because they add sensory credibility to the usual pressure tactics. A fake face or voice can create trust, authority, and urgency faster than text alone, which lowers the chance that people will pause and verify. That is why procedures must focus on request validation, not confidence in what is seen or heard.
Q: What are the signs that a disinformation or deepfake attack is succeeding?
A: Common signs include unusual urgency, pressure to bypass normal process, emotionally charged language, unexpected channel changes, and requests that depend on immediate trust in a familiar person. If a message demands fast action and discourages verification, the organisation should treat that as a control warning, not a convenience feature.
Q: How should organisations govern high-risk requests when synthetic media is possible?
A: They should require independent verification for payments, account changes, and sensitive approvals, and they should document who can approve what under what conditions. The goal is to make a convincing voice, face, or message insufficient on its own to trigger irreversible action.
Technical breakdown
How deepfakes turn trust into an attack surface
Deepfakes work because people are wired to trust what appears to be seen or heard directly. AI-generated images, video, and audio mimic familiar cues such as facial movement, voice tone, and timing, then combine them with urgency or authority to reduce scrutiny. The attack succeeds when the target treats the content as evidence instead of treating it as input that still requires verification. This is why deepfakes do not replace phishing or impersonation; they strengthen them by making the initial lie feel more real. For identity and fraud teams, the technical issue is not only content generation, but the collapse of trust in sensory confirmation.
Practical implication: move verification away from single-channel media cues and into authenticated, out-of-band confirmation paths.
Why emotion and bias improve attacker success
The article correctly highlights that deepfakes exploit cognitive shortcuts such as confirmation bias, social proof, authority bias, and urgency. These are not abstract psychology concepts in this context; they are control failures in human decision-making. When a message confirms what someone already expects, or appears to come from a respected leader, people are more likely to share it, act on it, or suspend normal checks. That is why disinformation campaigns often aim for rapid behavioural response rather than sustained deception. In security programmes, this means the risk is amplified whenever approval flows rely on speed, familiarity, or social pressure instead of stronger identity assurance.
Practical implication: design approval and escalation paths so that urgency never bypasses identity verification.
What makes deepfake detection incomplete as a defence
Visual inspection and content analysis still matter, but they are not sufficient as a primary control. The article notes that people overestimate their ability to detect manipulation, and even trained observers can miss synthetic artefacts when the material is convincing enough. That creates a governance gap: a control strategy built around human spotting assumes the defender sees the fake before any harm occurs. In practice, organisations need layered controls that verify source authenticity, validate requests through trusted channels, and limit the blast radius of any successful impersonation. The real lesson is that detection is useful, but resilience has to assume some forgeries will look believable.
Practical implication: pair detection with source verification, call-back procedures, and privilege-limiting approval controls.
Threat narrative
Attacker objective: The attacker aims to manipulate human trust into an action that creates financial, operational, or reputational harm.
- Entry occurs through a fabricated image, video, audio clip, or message that appears to come from a trusted person or event.
- Credential or trust access is gained when the target accepts the content as authentic and treats the request as legitimate.
- Escalation follows when urgency, authority, or social pressure pushes the target to approve payment, disclose information, or bypass normal checks.
- Impact is fraud, compromise, blackmail, misinformation spread, or organisational decision distortion.
Breaches seen in the wild
- Arup deepfake fraud 2024: Deepfakes of Arup's CFO and colleagues on a video call led a Hong Kong employee to transfer HK$200 million (about US$25.6m) to fraudsters.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Deepfake disinformation is now an identity risk, not just a media-verification problem. When a forged face or voice can stand in for a trusted executive, the control question shifts to how organisations confirm who is really making a request. That makes identity verification, out-of-band challenge, and approval hygiene part of the defence, not just communications policy. The practical conclusion is that identity assurance now has to extend into everyday human decision points.
Emotion is the attacker’s most reliable privilege escalation path. The article shows that fear, urgency, excitement, and social pressure override rational review more often than technical flaws do. That means the strongest disinformation campaigns do not need perfect synthesis, only believable timing and context. Practitioners should treat manipulation-resistant process design as a governance requirement, not a soft-awareness topic.
Trust frameworks need to be rebuilt around provenance, not appearance. Seeing, hearing, or reading something familiar no longer proves source legitimacy. In the broader fraud and identity verification landscape, this pushes organisations toward provenance checks, authenticated communications, and explicit escalation paths for high-risk requests. The field should stop assuming that human perception is an acceptable primary trust control.
Verification debt: the hidden exposure created when organisations rely on instinct instead of proof. Every process that assumes a person can spot a fake in time is accumulating governance debt. That debt becomes visible when an urgent request arrives through a convincing synthetic channel and the organisation has no reliable second factor for truth. The practitioner takeaway is to remove instinct from critical authorisation decisions wherever possible.
The liar’s dividend will complicate incident response and executive communications. As synthetic media becomes normalised, real evidence can be dismissed as fake and false claims can gain plausible deniability. That creates a second-order governance issue for legal, security, and communications teams because truth itself becomes contestable. The practical conclusion is to preserve signed, time-stamped evidence and rely on documented provenance for high-impact events.
From our research library:
- Deepfake scams now account for nearly $12 billion in fraud losses globally.
- Read next: Deepfakes, Social Engineering and AI Impersonation Guide
What this signals
Deepfake resilience is becoming part of identity assurance. Security teams should expect synthetic media to sit inside ordinary fraud and impersonation workflows, which means approval design matters as much as detection tooling. The right response is not to trust perception less in the abstract, but to rely on authenticated channels whenever a request can move money, data, or access.
The programme impact is clearest in finance, executive operations, and help desk workflows, where urgency and familiarity are easy to weaponise. Organisations that still allow high-risk actions to ride on a single voice, video, or message are leaving a trust gap that modern disinformation campaigns can exploit quickly.
For practitioners
- Implement out-of-band verification for high-risk requests Require callback or secondary-channel confirmation for payments, account changes, and urgent executive instructions, especially when the request is unusual or time-sensitive.
- Train staff on influence tactics, not just fake-media signs Teach teams to recognise urgency, authority, social proof, reciprocity, and confirmation bias so they can slow down before acting on suspicious content.
- Add provenance checks to executive and finance workflows Use approved contact directories, signing, and change-control rules so that a convincing voice or video cannot authorise material action by itself.
- Limit blast radius for impersonation-driven approvals Set tighter approval thresholds, dual-authorisation rules, and segregation of duties for requests that could create immediate financial or access impact.
- Preserve evidence for disputed media and decisions Keep timestamps, message headers, call records, and signed records so legal, security, and communications teams can investigate or refute synthetic claims.
Key takeaways
- AI-generated disinformation increases social engineering risk because it turns familiar-looking media into a vehicle for urgency, authority, and emotional pressure.
- The core weakness is not just media authenticity but human decision shortcuts that allow forged content to be treated as proof.
- Organisations should harden approval and verification processes so that convincing synthetic content cannot trigger irreversible action on its own.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATT&CK address the attack surface, NIST CSF 2.0 and NIST SP 800-63 set the technical controls, and GDPR defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI09 — Human-Agent Trust Exploitation | Synthetic media exploits human trust in convincing interactions, which maps closely to trust exploitation patterns. |
| Recommendation — Treat high-trust approval channels as attack surfaces and require independent verification before action. | ||
| NIST CSF 2.0 | PR.AA-05 — Access Permissions, Entitlements and Authorizations | The article's core issue is unauthorised action enabled by social engineering, which affects approval and authorisation. |
| Recommendation — Tighten authorisation checks so no single communication channel can trigger privileged action. | ||
| NIST SP 800-63 | SP 800-63C — Federation | Identity assurance and trusted assertions matter when people rely on external-looking communications and claimed identity. |
| Recommendation — Use stronger federation and verified identity assertions for requests that can change money or access. | ||
| GDPR | Art.32 — Security of Processing | Where deepfake-enabled fraud touches personal data or identity verification, processing security becomes relevant. |
| Recommendation — Protect identity and verification data with controls that reduce spoofing and impersonation risk. | ||
| MITRE ATT&CK | TA0001;TA0006 — Initial Access; Credential Access | The article describes impersonation leading to fraudulent action and trust compromise, both ATT&CK-relevant. |
| Recommendation — Map impersonation-driven fraud to initial access and credential access behaviours in your detections. | ||
Key terms
- Deepfake: Synthetic or altered media created with AI or machine learning so that a person appears to say or do something they never did. In security terms, deepfakes are trust attacks that can distort identity verification, approval workflows, and fraud detection.
- Disinformation: Disinformation is false or misleading content created with the intent to deceive. Unlike accidental misinformation, it is designed to manipulate beliefs or behaviour, making it a common tool in social engineering, fraud, and influence operations.
- Out-Of-Band Verification: A confirmation step that uses a different channel or method than the original request. It reduces the chance that a single spoofed email, voice call, or video session can authorize privileged activity or financial transfer.
- Provenance: Provenance is the traceable history of where a software artifact came from, who approved it, and what controls were applied along the way. In container security, provenance supports trust decisions because it links delivery steps to accountable identities and review points.
Deepen your knowledge
The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners strengthen identity controls across both human and non-human access paths.
Published by the NHIMG editorial team on August 2, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org