Generative AI lowers the cost and skill needed to create convincing messages, identities, and supporting evidence. That means attackers can scale social engineering, lure victims into authorising payments, and move money faster through crypto and real-time payment systems. Traditional identity checks alone are weaker because the user may be fully verified while the transaction itself is fraudulent.
How generative AI changes the scam-detection problem
Generative AI does not just make scams more persuasive, it changes the economics of fraud. Fraud teams now face higher-volume phishing, stronger pretexting, synthetic identities, cloned voices, and polished supporting documents that can look consistent enough to pass manual review. That raises the cost of verification because the visible content is no longer a reliable signal of legitimacy.
For banks, exchanges, and payment providers, the practical problem is that scam detection often depends on pattern recognition across language, identity, device, and transaction behaviour. When attackers can generate many variants quickly, they can probe controls, rotate narratives, and adapt faster than rule-based reviews or static fraud playbooks. This is one reason why NIST AI 600-1 GenAI Profile matters for organisations trying to govern provenance, testing, and incident handling around GenAI use.
It also means the fraud signal moves away from the message itself and toward the surrounding context. A convincing email, call, invoice, or KYC artifact may be entirely synthetic, but the real detection challenge is whether the request fits the customer’s normal behaviour, device history, beneficiary profile, and payment timing.
Why traditional identity checks are not enough
Traditional identity checks can succeed while the transaction is still fraudulent. A user may have passed KYC, MFA, or login verification, yet still be manipulated into authorising a transfer to an attacker-controlled destination. In those cases, the issue is not account takeover, it is authorised fraud enabled by social engineering.
This is especially hard for payments and exchanges because the system may see a legitimate session, a known device, and a valid customer. The fraud is hidden in intent, not access. That makes post-authentication controls important, including payment step-up, beneficiary risk scoring, cooling-off periods, and verification of high-risk behavioural anomalies. Where payment execution is involved, the relevant control logic often aligns with strong authorisation and least-privilege principles, such as those reflected in PCI DSS v4.0.
Generative AI increases the chance that the customer will trust the scammer long enough to complete the transfer. The better the fake support evidence, the less useful a single identity proof becomes as a fraud gate. Detection has to ask whether the transaction itself makes sense, not just whether the person is real.
Why speed, scale, and synthetic evidence make fraud easier to operationalise
Scams used to depend on time-consuming manual tailoring. Generative AI compresses that effort. Attackers can mass-produce tailored messages, image evidence, voice notes, fake support chats, and even multi-stage narratives that look consistent across channels. That improves conversion rates and lets fraud crews run more experiments until they find a working lure.
For payment providers, the danger is that faster fraud chains can outpace traditional intervention windows. In card, exchange, and real-time payment environments, a victim may be convinced, the payment may clear quickly, and the funds may be moved again before manual review completes. The same dynamic can appear in crypto flows, where velocity and irreversibility reduce the time available for intervention.
Defenders therefore need better correlation across communication, identity, device, and transaction telemetry. Detection that relies only on content inspection will miss the broader fraud pattern. That is why techniques and countermeasures from sources such as MITRE D3FEND and operational detection practices from SANS Security Resources are useful when teams need to convert broad scam signals into measurable controls.
Risk and Threat Considerations
Generative AI increases both the scale of fraud attempts and the realism of the evidence used to support them. That creates more opportunities for impersonation, payment redirection, and synthetic support artifacts to slip through review, especially where the defender assumes that a verified identity implies a legitimate transaction.
Failure mechanism: Attackers use GenAI to create persuasive pretexts, cloned communications, and fake evidence, then combine them with fast payment rails or crypto transfers before human review can intervene.
Impact: Organisations can see higher scam conversion, more authorised push payment losses, more false trust in synthetic artefacts, and weaker confidence in standard identity checks as a fraud control.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK addresses the attack surface, NIST AI 600-1, NIST SP 800-53 Rev 5 and NIST CSF 2.0 set the technical controls, and PCI DSS v4.0 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI 600-1 | Generative AI Profile | GenAI fraud content and provenance risks directly shape scam detection. |
| Recommendation — Apply GenAI provenance and incident controls to suspicious content workflows. | ||
| NIST SP 800-53 Rev 5 | IA-8 — Identification and Authentication (Non-Organizational Users) | Customer-facing scam detection depends on external-user identity verification. |
| Recommendation — Strengthen external-user authentication and step-up checks for risky transactions. | ||
| PCI DSS v4.0 | 7 — Restrict Access to System Components and Cardholder Data by Business Need to Know | Payment fraud controls hinge on limiting and reviewing high-risk transaction authority. |
| Recommendation — Restrict transaction capabilities to the minimum business need and review exceptions. | ||
| MITRE ATT&CK | T1566 — Phishing | GenAI scales phishing, pretexting, and social engineering used in scam delivery. |
| Recommendation — Map scam lures to phishing techniques and hunt for campaign infrastructure. | ||
| NIST CSF 2.0 | DE.CM-01 — Networks and systems are monitored to detect cybersecurity events | Banks and payment firms need monitoring that detects anomalous fraud behaviour. |
| Recommendation — Monitor transaction and communication telemetry for scam indicators and anomalies. | ||
Practitioner Guidance
What to prioritise: Shift scam detection from single-point identity verification to transaction-context verification. The highest-value controls are beneficiary checks, behavioural anomaly detection, and step-up review for unusual payment intent, not more friction at login.
What to verify: Treat the combination of device reputation, payment destination, narrative consistency, and historical customer behaviour as the detection unit. If those signals do not align, escalate even when the user is fully authenticated.
Practitioner takeaway: GenAI makes scams harder to spot because it improves the quality of the fraud story, so the decisive control is no longer “is this user real?” but “does this transaction make sense for this user right now?”
