AI-generated disinformation is reshaping social engineering risk

At a glance

What this is: This is an Anomali explainer on how AI-generated disinformation and deepfakes work, why they persuade people, and how individuals and organisations can identify fabricated content.

Why it matters: It matters to IAM, fraud, and trust-and-safety practitioners because deepfakes exploit verification gaps, override human judgement, and can bypass controls that assume people will pause, verify, and escalate suspicious requests.

By the numbers:

• In recent years, the volume of social engineering attacks, scams and fabricated media driven by deepfakes has increased significantly, with one analysis reporting a tenfold rise in deepfake related fraud between 2022 and 2023.
• A CDC backed meta-analysis across 56 studies involving more than 86,000 participants found average human accuracy for identifying deepfakes was only slightly above chance, typically between 57 and 60 percent.
• One early 2024 case used a live deepfake video call to impersonate a company’s CFO and trick an employee into transferring approximately £20 million GBP.

👉 Read Anomali's analysis of AI-generated disinformation and deepfakes

Context

AI-generated disinformation is a governance problem because it manipulates the human verification layer that many identity and security processes still depend on. In practice, the threat sits at the intersection of social engineering, fraud, and identity assurance, where confidence in what people see or hear can be more dangerous than a technical compromise.

The article is strongest when it explains that deepfakes do not need perfect realism to work. They need speed, emotion, and a believable context, which is why verification procedures that depend on intuition rather than out-of-band confirmation remain vulnerable. That makes this topic relevant to identity verification, IAM escalation paths, and any process that still treats human judgement as a sufficient control.

Key questions

Q: How should organisations handle identity verification when deepfakes can mimic real users?

A: Organisations should stop treating visual similarity as proof of identity and move high-risk workflows toward cryptographic verification, issuer trust, and device-bound proof of possession. Deepfakes are a scaling problem, so the control needs to fail closed when evidence can be simulated. Use selfies only where the business impact of a false accept is low.

Q: Why do deepfakes make social engineering more effective?

A: Deepfakes make social engineering more effective because they add sensory credibility to the usual pressure tactics. A fake face or voice can create trust, authority, and urgency faster than text alone, which lowers the chance that people will pause and verify. That is why procedures must focus on request validation, not confidence in what is seen or heard.

Q: What do organisations get wrong about spotting AI-generated disinformation?

A: They often assume that awareness alone is enough, or that people can reliably identify fakes by sight, sound, or instinct. In reality, detection is inconsistent and highly affected by emotion, context, and expectation. Organisations get better results when they build repeatable verification steps into process, rather than asking staff to improvise judgment under pressure.

Q: How can security teams reduce the impact of manipulated media?

A: Security teams can reduce impact by making manipulated media less useful to attackers. That means controlled approval paths, strict callback procedures, strong identity proofing, and incident playbooks that treat suspicious media as a trigger for verification rather than immediate action. The goal is to slow the decision until independent evidence supports it.

Technical breakdown

How deepfakes and disinformation bypass human verification

Deepfakes are synthetic images, audio, or video designed to appear authentic, while disinformation is false content created to mislead. Their effectiveness comes from exploiting cognitive shortcuts: people trust familiar faces and voices, overestimate their ability to spot fakes, and respond to urgency before checking context. This is why a convincing clip or voice note can succeed even when the content is imperfect. The article’s core technical point is that persuasion often matters more than realism; a fabricated message only needs to trigger an action. Practical implication: treat any unexpected request as untrusted until it is confirmed through a separate channel.

Practical implication: require out-of-band verification for any identity, payment, or access request that arrives through audio or video.

Why social engineering and deepfakes reinforce each other

Deepfakes do not replace classic social engineering, they amplify it. Attackers combine fabricated media with authority, scarcity, social proof, and emotional pressure to reduce the chance of challenge. A spoofed executive call works because the medium creates trust while the message creates urgency. The same pattern also applies to text-based disinformation, where AI-generated wording can be tuned to fit the audience’s beliefs and bias. In security terms, this is a trust-layer attack rather than a malware-first attack. Practical implication: train response teams to challenge the request path, not just the content of the message.

Practical implication: build escalation rules that force confirmation through a second trusted channel before money, credentials, or approvals move.

How to spot manipulated media without over-relying on tools

The article outlines a practical verification workflow: check whether multiple credible sources corroborate the claim, inspect the source account and context, and look for visual or audio anomalies such as distorted hands, mismatched lip movement, odd pacing, or unnatural background detail. It also recommends reverse image search and metadata review. These methods are useful, but they are not perfect because high-quality synthesis is improving quickly. The real control is procedural discipline. Tools help, but they do not replace a habit of deliberate verification. Practical implication: standardise a simple, repeatable verification checklist for staff who may receive manipulated content.

Practical implication: make verification steps mandatory in incident, fraud, and executive communication playbooks.

Threat narrative

Attacker objective: The attacker wants the target to make a high-trust decision based on fabricated evidence, enabling fraud, influence, or access abuse.

1. Entry occurs through a fabricated image, video, audio clip, or message that appears credible enough to trigger attention or trust.
2. Escalation happens when the victim accepts the apparent authority or urgency and bypasses normal verification.
3. Impact follows when the victim shares false information, transfers funds, or authorises an action the attacker wanted.

NHI Mgmt Group analysis

AI-generated deception is now a trust-verification problem, not just a content-moderation problem. Deepfakes matter because they exploit the assumptions behind human approval workflows, especially where identity is established through sight or sound. That creates governance exposure in finance, HR, support, and executive operations. Practitioners should treat manipulated media as a control failure in verification design, not merely a user awareness issue.

Deepfake defence belongs in identity assurance, fraud operations, and access governance. The article shows that the same manipulative patterns can be used to approve payments, influence decisions, or impersonate authority. That makes this a cross-functional problem for IAM, verification, and SOC teams, because the attack path often begins outside the technical perimeter. Organisations need verification rules that are explicit, auditable, and consistent across channels.

Verification trust gap: the critical weakness is the gap between what a person believes is authentic and what the organisation can actually prove. Once that gap exists, attackers only need one persuasive interaction to bypass policy. This is especially relevant where identity proofs, executive approvals, or step-up checks still depend on human judgement. Practitioners should close the gap with procedural controls, not confidence in recognition ability.

Human resistance to manipulation is not a reliable security control. The article is a reminder that emotion, bias, and urgency are exploitable conditions, not edge cases. Awareness training helps, but it is too weak on its own when a convincing fake can arrive at machine speed. Security and fraud programmes should assume that people will sometimes believe the wrong thing and build containment around that reality.

Identity verification must be designed for adversarial media. As AI-generated audio and video become easier to produce, organisations need verification methods that are independent of the medium being used. That means stronger callback procedures, layered approval rules, and clear escalation paths when a request is unusual. In practice, the safest model is to make synthetic media irrelevant to the decision.

What this signals

Verification trust gap: organisations will need to treat synthetic media as a routine input to fraud, IAM, and security operations rather than a fringe threat. That means codifying challenge steps, callback rules, and approval boundaries so that executive authority cannot be simulated into existence.

The practical signal for programmes is that identity assurance now has to survive adversarial content, not just credential abuse. If your approvals, resets, or exceptions can be triggered by a convincing call or clip, the control set is too dependent on human perception and not enough on independent proof. Use the CISA cyber threat advisories for broader threat context and keep verification steps anchored to process, not instinct.

For practitioners

• Implement out-of-band verification for high-risk requests Require a second trusted channel before approving payments, password resets, privileged access, or executive requests that arrive by voice, video, or message. Use known contact details and never rely on the inbound channel itself.
• Add deepfake scenarios to fraud and security playbooks Test how teams respond when a fake executive call, synthetic customer video, or AI-generated threat message is used to pressure staff into action. Measure whether escalation and challenge steps actually happen under time pressure.
• Standardise content verification checks Create a simple workflow that asks staff to check source reputation, context, corroborating reports, and visual or audio anomalies before sharing or acting on suspicious material. Make the process easy enough to use during incidents.
• Strengthen approval controls for identity-dependent decisions Separate request initiation from approval wherever possible, especially for financial transfers, identity proofing exceptions, and privileged changes. Require independent confirmation for any action that would be hard to reverse.

Key takeaways

• AI-generated disinformation turns human perception into an attack surface, especially where authority and urgency drive decisions.
• The most serious failures happen when organisations trust what they hear or see instead of requiring independent verification.
• Procedural controls, not confidence in human detection, are what reduce the impact of deepfakes and fabricated media.

Key terms

• Deepfake: Synthetic or altered media created with AI or machine learning so that a person appears to say or do something they never did. In security terms, deepfakes are trust attacks that can distort identity verification, approval workflows, and fraud detection.
• Disinformation: Disinformation is false or misleading content created with the intent to deceive. Unlike accidental misinformation, it is designed to manipulate beliefs or behaviour, making it a common tool in social engineering, fraud, and influence operations.
• Out-Of-Band Verification: A confirmation step that uses a different channel or method than the original request. It reduces the chance that a single spoofed email, voice call, or video session can authorize privileged activity or financial transfer.
• Liar's dividend: The liar's dividend is the benefit gained when the existence of deepfakes makes real evidence easier to dismiss as fabricated. It weakens accountability because genuine footage, audio, or documents can be challenged simply by claiming they are synthetic.

What's in the full article

Anomali's full post covers the practical detection methods and reference resources this analysis intentionally leaves at a higher level:

• Detailed visual and audio anomaly checks for deepfake images, audio, and video that can be used in incident triage.
• Step-by-step guidance on using reverse image search and metadata review to validate suspicious media before acting.
• A structured verification process for handling suspicious messages, calls, and clips in security and fraud workflows.
• A curated set of external resources for fact checking, forensic review, and media verification.

👉 Anomali's full post covers detection checks, verification workflow, and practical online resources.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course, the industry's only accredited NHI security programme. It is designed for practitioners building stronger governance across identity, access, and lifecycle control.