Large language models let attackers write fluent, personalised phishing in any language at scale, which removes the spelling and grammar mistakes that awareness training taught people to look for. The UK NCSC assessed in January 2024 that generative AI would make it hard for people at any level of security awareness to judge whether an email or password reset request is genuine, and the FBI warned in December 2024 that criminals use AI translation and writing tools to remove the errors that would otherwise expose them. This guide explains how AI changes phishing, where it shows up, and what defenders should change in detection, email authentication, identity controls and training. It is for email security, SOC, identity and awareness teams. The wider picture of AI-enabled attacks is in the AI-Powered Cyber Attacks Guide; this page focuses on phishing defence.
Key takeaways
- Language quality is no longer a signal. Retire training that teaches staff to spot bad grammar, and teach them to check the request, the channel and the sender instead.
- AI changes scale, not the kill chain. Google's threat intelligence found actors use generative AI for productivity, not novel capability: lures, translation and research. The same controls still break the chain, provided they do not depend on humans spotting the lure.
- Make credentials unphishable. Phishing-resistant MFA (FIDO2 security keys, passkeys, certificate-based authentication) defeats credential phishing however convincing the lure is.
- Verify actions out of band. Payment changes, MFA resets and access grants need a callback on a known number or a second approver, because voice and video can now be faked too.
- Detect on behaviour and infrastructure. Sender history, domain age, authentication results, unusual requests and post-click identity signals catch AI lures that content filters miss.
How the attack works
AI-generated phishing follows the familiar sequence. What changes is the cost and quality of each step. MITRE ATT&CK records the attacker's use of AI tools as T1588.007 Obtain Capabilities: Artificial Intelligence, and MITRE ATLAS describes LLM-driven social engineering as AML.T0052.000 Spearphishing via Social Engineering LLM and deepfake-assisted phishing as AML.T0052.001.
| Stage | What AI adds | ATT&CK / ATLAS | Where defenders can break it |
|---|---|---|---|
| Reconnaissance | Fast summaries of a target's role, suppliers, projects and writing style from public sources | T1593 Search Open Websites/Domains; T1589 Gather Victim Identity Information | Limit public detail on finance and admin roles; monitor brand and executive impersonation |
| Lure writing | Fluent, localised, context-aware text in many variants; no spelling or grammar errors | T1588.007; AML.T0052.000 | Content filters weakened; rely on sender, infrastructure and request signals |
| Infrastructure | AI-written landing pages and chatbots on fake sites that steer victims to links | T1585.002 Establish Accounts: Email Accounts | Domain age and lookalike detection, DMARC enforcement on your own domains |
| Delivery | Email, SMS, chat and social platforms; voice and video deepfakes for follow-up | T1566.001, T1566.002, T1566.003, T1566.004; AML.T0052.001; AML.T0088 Generate Deepfakes | Gateway, URL rewriting and detonation; verification of voice and video requests |
| Conversation | Real-time replies that keep a victim engaged across several messages | T1598 Phishing for Information | Policy that sensitive actions never complete inside the conversation |
| Payoff | Credentials, MFA approval, payment, or a malicious link click | T1204.001 Malicious Link | Phishing-resistant MFA, payment controls, endpoint protection |
The FBI's December 2024 alert (I-120324-PSA) lists the practical uses: large numbers of fake social profiles, faster and more convincing messages, translation that removes tell-tale errors, AI chatbots on fraudulent websites, forged identity documents, cloned voices and real-time video calls impersonating executives or authority figures.
Where it shows up
- Business email compromise: invoice and payment change requests written in a supplier's or executive's style, often with no link or attachment for filters to catch. See the Email Identity and BEC Guide.
- Credential phishing: convincing password reset, document share and MFA re-enrolment emails that lead to adversary-in-the-middle proxies. See the AiTM Phishing Guide.
- Help desk and IT support pretexts: AI-scripted calls and chats asking for MFA resets or new device enrolment, followed by voice cloning where a known voice helps.
- Internal spearphishing: after one mailbox is compromised, AI drafts replies inside real threads (ATT&CK T1534 Internal Spearphishing), which inherit the trust of the conversation.
- Recruitment and social platforms: fake recruiter or candidate personas with AI profile photos and messages, a pattern Google's threat intelligence group associated with North Korean IT worker activity.
- Multilingual campaigns: targets in languages attackers do not speak, which the NCSC and FBI both identify as a direct benefit of AI translation.
Impact
- Higher success for less-skilled actors: the NCSC rated the uplift to social engineering and phishing for less-skilled criminals as significant (from a low base), so expect more competent campaigns from more actors.
- Ransomware access: the NCSC notes phishing often provides initial access for ransomware, so better lures feed ransomware.
- Direct financial loss: payment fraud and BEC, where one convincing request can move large sums.
- Erosion of trust signals: staff can no longer rely on the look and sound of a message, a call or a video, which slows legitimate work unless verification is designed into processes.
- Training debt: awareness programmes built around spotting errors now teach the wrong lesson.
Detection
Detection has to move from "does this look like phishing" to "is this sender, infrastructure and request normal for this recipient".
| Signal | Source | Why it survives AI-written text |
|---|---|---|
| First-time or rare sender to this recipient, display name matching an executive or supplier | Email gateway, message trace | Fluency does not change who sent the message |
| SPF, DKIM and DMARC results and alignment; lookalike or newly registered domains | Gateway headers, DNS and domain intelligence | Infrastructure still has to exist and be authenticated or not |
| Reply-to differing from sender, or a new external forwarding rule after a reply | Message trace, mailbox audit | A common BEC mechanic independent of the lure |
| Request type: bank detail change, urgent payment, MFA reset, gift cards, credentials | Gateway natural language classification, finance workflow | The ask is the attack; classify on intent, not style |
| Post-click identity events: sign-in from new infrastructure, token replay, new MFA method | Identity provider sign-in and audit logs | Catches the outcome even when the email was delivered |
| User reports | Report button, triage queue | People still notice odd requests even when the writing is perfect |
Pair gateway detections with message trace hunting, see the Email Gateway and Message Trace Detection Guide, and make the report button fast to triage, see the Phishing Report Button and User Reported Message Triage Guide.
Metrics that show it is working:
- Share of workforce sign-ins using phishing-resistant MFA, and the number of accounts still on SMS or push.
- Median time from first user report to removal of the campaign from all mailboxes.
- Reporting rate in simulations that use fluent, personalised lures (not error-laden templates).
- Percentage of payment detail changes verified through an independent channel before payment.
- DMARC enforcement (
p=quarantineorp=reject) across all sending and non-sending domains you own.
Prevention and hardening
Identity controls first
- Phishing-resistant MFA: FIDO2 security keys, passkeys and certificate-based authentication bind the login to the real site, so a perfect lure still yields nothing usable. Prioritise administrators, finance, help desk and executives. See the Phishing-Resistant MFA Guide.
- Help desk verification: MFA resets and device enrolments need identity verification that a voice or video cannot satisfy on its own, such as a callback to a number from the HR record and manager approval. See the Deepfake-Resistant Identity Verification Guide.
- Token and session protection: conditional access on device compliance and token protection reduce the value of a stolen session after an AiTM click.
Email and domain controls
- DMARC on your domains: publish SPF and DKIM, then move DMARC from
p=nonetoquarantineorreject, including non-sending domains. DMARC is now specified in RFC 9989 (Proposed Standard, May 2026), which obsoletes RFC 7489 and removes thepcttag, so review any staged rollout that relied onpct. - Inbound enforcement: honour sender DMARC policies, tag external mail, flag first-contact senders and display name impersonation of executives, and block or warn on newly registered lookalike domains.
- Link and attachment handling: URL rewriting with time-of-click checks and detonation of attachments (NIST SP 800-53 SC-44 Detonation Chambers).
Process controls
- Payment verification: any change of bank details is confirmed by calling a number already on file, never one in the message, with dual approval above a threshold. See the Business Email Compromise Awareness for Finance Teams Guide.
- Code words for high-risk requests: the FBI recommends a pre-agreed secret word or phrase to verify identity; organisations can apply the same idea to executive payment requests.
Awareness that still works
- Teach people to judge the request (money, credentials, MFA, secrecy, urgency) and the channel, not the writing.
- Run simulations with fluent, personalised lures and multi-step conversations, and measure reporting rather than click-shaming. See the Phishing Simulation Programme Guide.
- Include voice and video scenarios for finance, HR and help desk staff.
Rollout order
- Inventory high-risk roles and their MFA methods; start phishing-resistant MFA with admins and finance in a pilot ring, keeping a recovery path for lost keys.
- Write the out-of-band verification rule for payments and MFA resets, test it with a tabletop, then enforce it.
- Move DMARC to enforcement domain by domain after reviewing aggregate reports (RFC 9990 format) for legitimate senders; roll back a domain to
p=noneif legitimate mail fails. - Tune gateway impersonation and first-contact rules in audit or tag-only mode for two to four weeks before quarantining.
- Refresh training and simulations to the new lure style, then measure the metrics above each quarter.
Framework mapping
| Control | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 |
|---|---|---|
| Awareness on social engineering | AT-2 Literacy Training and Awareness; AT-2(3) Social Engineering and Mining; AT-3 Role-based Training | PR.AT-01 |
| Phishing-resistant authentication | IA-2 Identification and Authentication (Organizational Users) | PR.AA-05 |
| Email filtering and detonation | SI-8 Spam Protection; SC-44 Detonation Chambers; SI-3 Malicious Code Protection | DE.CM-09 |
| Analysis of reports and alerts | SI-4 System Monitoring; IR-4 Incident Handling | DE.AE-02 |
Lessons from incidents
The Arup deepfake fraud shows where AI-generated phishing ends up. A message claiming to be from the UK-based chief financial officer asked a Hong Kong finance worker for a secret transaction, and a video conference in which everyone else was a deepfake of real colleagues, built from public footage, removed the worker's doubts. HK$200 million was sent in 15 transfers. No content filter would have stopped it: what would have stopped it was a rule that high-value transfers are verified through a separate, pre-registered channel, whatever the requester looks and sounds like. For response steps, see the Deepfake CEO Fraud Response Playbook.
Common failure patterns:
- Training and simulations still built around spelling mistakes and obvious tells.
- MFA based on push or SMS for the very roles attackers target, so a convincing lure still captures a working login.
- Verification steps that use the contact details in the suspicious message.
- DMARC left at
p=nonefor years, so attackers can send as your own domain to your suppliers. - Help desk resets granted on a convincing voice alone.
Practitioner checklist
- Remove "look for bad grammar" from awareness content and replace it with request and channel checks.
- Roll out phishing-resistant MFA to admins, finance, help desk and executives first.
- Enforce out-of-band verification for bank detail changes, payments and MFA resets.
- Move every owned domain to DMARC
quarantineorrejectand review RFC 9989 changes. - Enable first-contact, display name impersonation and lookalike domain detections on the gateway.
- Correlate user reports and gateway alerts with identity sign-in events after clicks.
- Run simulations with fluent, personalised and multi-step lures, including voice for high-risk teams.
- Track the five metrics above quarterly and report them to leadership.
Standards and references
- NCSC: The Near-Term Impact of AI on the Cyber Threat
- FBI IC3: Criminals Use Generative Artificial Intelligence to Facilitate Financial Fraud
- Google Threat Intelligence Group: Adversarial Misuse of Generative AI
- CISA, NSA, FBI and MS-ISAC: Phishing Guidance, Stopping the Attack Cycle at Phase One
- IETF: RFC 9989 Domain-Based Message Authentication, Reporting, and Conformance (DMARC)
- MITRE ATT&CK: Phishing
- MITRE ATLAS: Phishing
- NIST: SP 800-53 Rev. 5
Related NHI Mgmt Group resources: AI-Powered Cyber Attacks Guide · Phishing Guide · Deepfake and AI Impersonation Guide · Phishing-Resistant MFA Guide