Generative AI Fraud: How Criminals Use AI, How to Defend

Generative AI has made convincing fake documents, synthetic identities and fluent scam conversations cheap enough to produce at scale, which removes many of the cues that fraud controls and customers used to rely on. Spelling mistakes, poor photo edits and a script that falls apart under questioning are no longer reliable warning signs. This guide covers the main ways criminals use generative AI across the fraud lifecycle: AI-generated identity documents and synthetic identities at onboarding, LLM-written scam scripts and chatbots for romance, investment and phishing fraud, cloned voices for account access, and automation of fraud operations. It then sets out the controls, detection signals and rollout steps that still work. It is for fraud, financial crime and identity teams in banks, fintechs, marketplaces and other digital businesses. Real-time deepfake impersonation of executives and staff is covered in depth in the Deepfake and AI Impersonation Guide; this page covers the wider fraud picture.

Key takeaways

AI attacks the evidence, not the system. Most AI-enabled fraud defeats checks that rely on how a document, voice or message looks or sounds, so move trust to signals that are hard to fake.
Onboarding is the front line. FinCEN's 2024 alert describes AI-altered or wholly synthetic ID images combined with stolen or fabricated personal data to open accounts that then launder fraud proceeds.
Scams now scale without skilled staff. LLMs write fluent messages in any language, run fake profiles and power chatbots on fraudulent investment sites.
Detection tools help but do not decide. NIST's synthetic content report finds that no single detection or watermarking technique is a comprehensive solution, so layer them with process controls.
Out-of-band verification still wins. A call-back to a known number, a pre-agreed family phrase or a second approver defeats most AI impersonation regardless of how good the fake is.

What generative AI changes in fraud

Generative AI does not create new fraud types so much as remove the costs and skills that limited old ones. The FBI's December 2024 public service announcement (I-120324-PSA) lists how criminals use AI-generated text, images, audio and video, and Europol's 2023 report on large language models flagged fraud, social engineering, disinformation and low-skill malicious code as the main criminal uses, warning that "dark LLMs" built to produce harmful output could become a criminal business model. Underground tools marketed under names such as WormGPT and FraudGPT were widely reported from 2023 onwards, but jailbroken mainstream models and open-weight models do much of the same work.

Volume: one operator can run thousands of tailored conversations or fake profiles at once.
Quality: the FBI notes that translation and writing tools remove the grammar and spelling errors that used to expose foreign scam operations.
Personalisation: public posts, breached data and voice or video clips are turned into messages, voices and faces tailored to one victim.
Speed of iteration: when a document check or script stops working, a new variant can be generated in minutes.

A typology of AI-enabled fraud

AI capability	Fraud uses (FBI, FinCEN)	Where it hits	Controls that still work
Text generation and translation	Phishing, fake social profiles, investment site content, scam chatbots, fake reviews	Customers, staff, marketplaces	Domain and sender authentication, payee checks, behavioural scam detection, customer warnings in the payment journey
Image generation and editing	Fake driver's licences and passports, synthetic profile photos, fake credentials, sextortion images, fake disaster photos for charity fraud	Onboarding, KYC, hiring, donations	Document authenticity checks beyond the image, data source verification, injection attack detection, liveness
Voice cloning	Family emergency scams, impersonating customers to bank contact centres	Contact centres, consumers	No voice-only authentication, call-back to number on file, family safe word
Video generation, real time	Fake executives or officials on video calls, fake verification selfies, promotional videos for investment scams	Payments approval, KYC, hiring	Out-of-band confirmation, dual approval, liveness with injection detection
Automation and code	Scripted account creation, form filling, testing stolen data at scale	Sign-up, login, checkout	Bot management, velocity rules, device intelligence

AI-generated documents and synthetic identities

FinCEN's alert FIN-2024-Alert004 (13 November 2024) describes criminals altering or generating images for driver's licences and passport cards, either modifying an authentic photo or creating a synthetic one, and combining them with stolen or entirely fabricated personal data to create synthetic identities. Those identities pass identity verification and customer due diligence, then receive and launder proceeds, including as funnel accounts. Synthetic identity fraud predates generative AI; what has changed is the realism and cost of the supporting documents. See the Synthetic Identity Fraud Guide.

FinCEN red flag	What it suggests	Practical control
Photo shows signs of alteration or does not match other details such as date of birth	Edited or generated ID image	Document forensics plus cross-check of extracted data against authoritative sources
Multiple identity documents conflict	Assembled synthetic identity	Consistency checks across all submitted documents and data
Third-party webcam plugin used during live verification	Video injection	Virtual camera and injection attack detection; trusted capture SDK
Customer switches channel or reports repeated glitches during verification	Avoiding a check the fake cannot pass	Do not downgrade to a weaker method; route to manual review
Customer declines multifactor authentication	Account intended for short-term misuse	Make MFA enrolment a condition of higher limits
Reverse image search matches a gallery of AI-generated faces	Generated face	Image similarity checks against known synthetic galleries and prior applications
Device or location data inconsistent with the documents	Remote or mule-operated account	Device and network intelligence at onboarding
New account with rapid transactions, high-risk payees, or high chargebacks	Funnel or mule account	Early-life account monitoring. See the Money Mule Detection Guide.

Move away from image-only checks: chip reading of passports and identity cards (NFC), verification against issuing or authoritative data sources, and bank or telecom data checks are far harder to fake than a photo. See the Identity Proofing and KYC Guide.
Protect the capture channel: the strongest liveness check is defeated if a generated video is injected into the stream. See the Biometric Liveness and Presentation Attack Detection Guide and the Deepfake-Resistant Identity Verification Guide.
Other fake documents: bank statements, payslips, utility bills and invoices are just as easy to generate. Verify income and address through open banking, payroll or utility data where possible rather than uploaded PDFs.
Hiring: generated CVs, profile photos and real-time face swaps in interviews are a growing route for insider access. See the Hiring Fraud and Candidate Identity Verification Guide.

LLM scam scripts and conversational fraud

Romance, investment, job and impersonation scams depend on long, persuasive conversations. LLMs let criminals hold many of them at once, in fluent language, with consistent personas. The FBI lists fake social media profiles, faster and more believable messages, AI-generated content for cryptocurrency investment websites and chatbots embedded on fraudulent sites as current uses.

Persona farms: generated profile photos and backstories across social media, dating and messaging platforms. Platform teams should combine image similarity, account creation velocity and network signals rather than judging single profiles. See the Fake Account Creation and Sign-up Fraud Guide.
Fake investment platforms: professional-looking sites with AI-written content, fake testimonials and support chatbots. Banks see the result as authorised push payments to new crypto or overseas payees. See the Authorised Push Payment (APP) Fraud Guide.
Scripted social engineering of staff: polished emails and chat messages that mimic a supplier's or executive's style for invoice redirection and payroll changes. See the Invoice Redirection and Supplier Bank Detail Fraud Guide.
Customer-side signals: because the victim authorises the payment, detection depends on behaviour: a first payment to a new payee after a long session, unusual amounts, coaching indicators such as remote access tools running, or a customer reading out answers. Dynamic, specific warnings at the moment of payment work better than generic ones.

Voice cloning and impersonation for account access

The FBI warns that criminals use short cloned clips to imitate a loved one in a fake crisis and to impersonate individuals to gain access to bank accounts. In February 2024 the US Federal Communications Commission ruled (FCC 24-17) that AI-generated voices, including voice cloning, count as "artificial" voices under the Telephone Consumer Protection Act, so robocalls using them need prior express consent. That gives enforcement a hook, but it does not stop criminal calls.

Retire voice as an authenticator: voice biometrics or "recognising the customer" should never be a sole factor for account changes or payments in a contact centre. See the Voice AI Agent Security and Fraud Guide.
Protect account changes: phone number, email, address and payee changes requested by phone need in-app confirmation or a call-back to details on file. See the Account Recovery and Help Desk Security Guide.
Executive and staff impersonation: the Arup deepfake fraud of January 2024 is the reference case: a finance worker made 15 transfers totalling HK$200 million after a video call in which every other participant was a deepfake. No system was breached; the control that failed was verification of the request.

Detecting AI-enabled fraud

NIST AI 100-4 (November 2024) reviews provenance (watermarks and metadata such as C2PA content credentials) and synthetic content detection. Its findings are sobering for anyone hoping to buy a single detector: detectors often work well only on the generators they were trained on, accuracy falls after compression or resizing, text detectors perform near chance on short texts, most watermarks can be removed, and spoofing can make unwatermarked content look watermarked. NIST concludes that none of these techniques offers a comprehensive solution on its own.

Control layer	Purpose	Strength against AI fraud
Process controls (call-back, dual approval, payee confirmation)	Verify the request, not the media	High: independent of how good the fake is
Authoritative data checks (NFC chip, issuer and bank data)	Verify identity claims against sources	High for onboarding
Phishing-resistant MFA and device binding	Bind access to a device the attacker does not hold	High for account access
Injection and liveness detection	Confirm a live person on a genuine camera	Medium to high, needs regular testing
Content detection and provenance	Flag likely synthetic media	Medium: useful signal, high error rate on new generators
Behavioural and network analytics	Spot scripted, coached or ring behaviour	Medium to high, depends on data

Metrics

Onboarding fraud rate: confirmed synthetic or fake-document accounts per 10,000 approvals, measured at 90 and 180 days.
Injection attempts detected: virtual camera or stream manipulation attempts per week, and the share that reached manual review.
Verification bypass rate: high-value payment or account change requests executed without the required out-of-band check.
Scam payment interventions: share of flagged authorised payments stopped or delayed after a warning, and confirmed scam losses per million payments.
Red team pass rate: share of internal synthetic document and deepfake tests caught by current controls.

How to implement a defence programme

Map exposure: list every journey that trusts how something looks or sounds: document upload, selfie checks, voice authentication, video approvals, email-based payment instructions.
Fix process controls first: introduce call-back to known numbers and dual approval for high-value payments and payee changes; this is cheap and defeats most impersonation.
Strengthen onboarding: add authoritative data checks and NFC chip reading, injection detection and device intelligence; pilot on a share of traffic and compare fraud and pass rates before full rollout.
Retire weak authenticators: phase out voice-only and knowledge-based authentication for account changes, with a fallback path for customers who cannot use the new method.
Add detection as a signal: run synthetic media detection in monitoring mode first, measure false positives, then use it to route cases to review rather than to auto-decline.
Red team regularly: generate synthetic documents, voices and videos against your own controls under an agreed test plan, and track the pass rate.
Report and share: US institutions filing suspicious activity reports on these schemes should include the key term FIN-2024-DEEPFAKEFRAUD in SAR field 2 and the narrative, as FinCEN requests. Share patterns through fraud consortiums.
Educate customers and staff: the FBI recommends a family secret word, hanging up and calling back on a number you look up yourself, and limiting public voice and image content.

Framework mapping

Area	Reference
Identity proofing at onboarding	NIST SP 800-53 IA-12 Identity Proofing; NIST CSF 2.0 PR.AA-02
Authentication of users and staff	NIST SP 800-53 IA-2
Adversary use of AI tooling	MITRE ATT&CK T1588.007 Obtain Capabilities: Artificial Intelligence
Fake personas and phishing	MITRE ATT&CK T1585.001 Social Media Accounts, T1566 Phishing, T1598 Phishing for Information
Generative AI risk management	NIST AI 600-1 Generative AI Profile. See the NIST AI 600-1 Generative AI Profile Guide.
Suspicious activity reporting (US)	FinCEN FIN-2024-Alert004, key term FIN-2024-DEEPFAKEFRAUD
AI voice robocalls (US)	FCC 24-17 Declaratory Ruling under the TCPA

Common failure patterns

Trusting the image: onboarding relies on a photo of an ID and a selfie, with no data source check and no injection detection.
Buying a detector as the control: a deepfake or AI text detector is set to auto-decline, then fails on a new generator while blocking genuine customers.
Downgrading on failure: when a verification step "glitches", the journey falls back to a weaker method, exactly what FinCEN lists as a red flag.
Voice or face as authority: staff act on a familiar voice or face on a call, as in the Arup case, instead of a verified request.
Ignoring early-life accounts: synthetic accounts pass onboarding, then behave like mules within days while monitoring treats them as normal new customers.
Generic warnings: customers see the same scam warning on every payment and learn to click through it.

Practitioner checklist

Inventory every journey that trusts appearance or voice and assign an owner to each.
Require call-back to known numbers and dual approval for high-value payments and payee or contact detail changes.
Add authoritative data checks, NFC chip reading and injection attack detection to onboarding.
Stop using voice or knowledge-based answers as a sole authenticator for account changes.
Monitor new accounts closely for mule and funnel behaviour in their first 90 days.
Run synthetic media detection as a routing signal, measured for false positives, not as an automatic decision.
Red team your controls with synthetic documents, cloned voices and deepfake video at least twice a year.
Use the FinCEN key term in relevant US SARs and share patterns through consortiums.
Give customers specific, in-journey scam warnings and promote safe words and call-back habits.

Generative AI Fraud Guide: How Criminals Use AI and How to Defend