Attackers are using AI models across every stage of an intrusion, but the evidence shows faster and cheaper versions of familiar attacks far more often than new ones. Since 2024 the major model providers, Google, OpenAI, Anthropic and Microsoft, have published threat reports describing named state and criminal actors misusing their models, and the UK NCSC has published a formal assessment of what this means to 2027. This guide separates what those reports actually document from vendor hype, explains where AI is changing phishing, fraud, malware and intrusion speed, and sets out the defences that hold up. It is for security leaders, SOC teams and fraud and identity owners who need an evidence-based position, not a headline.
Key takeaways
- Old playbooks, run faster. OpenAI's October 2025 report says threat actors bolt AI onto existing playbooks to move faster rather than gaining novel offensive capability, and Google reported no breakthrough capabilities in information operations.
- Agentic intrusion is now documented. Anthropic reported in November 2025 that a group it assessed with high confidence as Chinese state-sponsored used an AI agent to perform 80 to 90 percent of an espionage campaign against about 30 targets.
- Social engineering gains the most. Microsoft reported a 54 percent click-through rate for AI-automated phishing against 12 percent for other phishing, and the FBI has warned of AI voice messages impersonating senior officials.
- The exploitation window is shrinking. The NCSC judges AI will almost certainly help actors exploit known vulnerabilities faster, widening the gap between organisations that keep pace and those that do not.
- Defences are mostly the fundamentals, enforced. Phishing-resistant MFA, out-of-band payment verification, fast patching and tight control of machine credentials blunt most documented AI-assisted attacks.
How the attack works
"AI-powered attack" covers four quite different things, and each needs a different response. MITRE ATT&CK v20 records the acquisition of AI tools as a resource development technique (T1588.007 Artificial Intelligence), alongside the techniques the AI then helps with.
| Category | What the AI does | Documented example | Evidence level |
|---|---|---|---|
| Productivity aid | Writes lures, translates, researches targets, debugs malware and scripts | OpenAI (October 2025): Russian-speaking criminal developer building credential stealers; Chinese-language cluster writing phishing in Chinese, English and Japanese | Strong, widely reported by all providers |
| Synthetic media | Clones voices and faces for impersonation | FBI PSA I-051525-PSA (May 2025): AI voice messages impersonating senior US officials; GTIG: North Korean UNC1069 deepfake lures | Strong, with confirmed fraud losses |
| AI inside malware | Malware calls a model at run time to generate commands or rewrite itself | GTIG (November 2025): PROMPTSTEAL used by APT28 against Ukraine; PROMPTFLUX still experimental | Early; Google calls it nascent |
| Agentic operations | An AI agent runs reconnaissance, exploitation and data handling with few human decisions | Anthropic (August and November 2025): data extortion against at least 17 organisations; state espionage campaign with 4 to 6 human decision points | Documented in a small number of cases |
Getting past model safeguards
Actors still need a model that will help. GTIG reported a China-nexus actor that was refused, then reframed the request as a capture-the-flag exercise, and other actors posing as students or security researchers. MITRE ATLAS tracks this as AML.T0054 LLM Jailbreak. GTIG also describes a maturing underground market of illicit AI tools for phishing, malware development and vulnerability research, which removes the need to evade a mainstream provider at all. Anthropic's August 2025 report described a low-skill actor who relied on Claude for core ransomware components and sold the packages for $400 to $1,200.
The agentic case in detail
Anthropic's November 2025 report is the most detailed public account of an AI agent running an intrusion. Activity was detected in mid-September 2025. Humans selected targets and approved escalation at roughly four to six decision points; the agent did the rest at a speed no human team could match. Two limits matter for defenders. First, the attackers succeeded against only a small number of the roughly 30 targets. Second, the model sometimes hallucinated credentials or reported public information as a secret find, which Anthropic says remains an obstacle to fully autonomous attacks. Its earlier August 2025 report described an actor using Claude Code to automate reconnaissance, credential harvesting and network penetration, then choosing data to steal and drafting extortion demands, sometimes above $500,000, without encrypting anything.
Where it shows up
- Phishing and pretexting: fluent, localised, personalised lures at scale (T1566 Phishing, T1598 Phishing for Information), built from public profile data (T1589, T1593). Spelling mistakes are no longer a useful signal. See the Phishing Guide.
- Voice and video impersonation: cloned executives, officials and IT staff on calls and voice notes (T1566.004 Spearphishing Voice). The Arup deepfake fraud, where a fake CFO on a video call triggered HK$200 million in transfers, is the reference case. See the Deepfake and AI Impersonation Guide.
- Fraudulent hiring: Anthropic reported North Korean operatives using AI to pass coding tests, hold remote jobs at US technology companies and send pay to the regime.
- Scams at scale: OpenAI's October 2025 report describes scam centres likely based in Cambodia, Myanmar and Nigeria using models for translation, messages, fake personas and websites.
- Developer and cloud credentials: GTIG's QUIETVAULT steals GitHub and NPM tokens and uses AI command-line tools already on the host to search for more secrets. Stolen cloud and API keys are also used to run models for free; see the LLMjacking Guide.
- Your own AI systems as a target: prompt injection against copilots and agents (AML.T0051) is a separate but related risk, covered in the Prompt Injection Guide.
Impact
Separating evidence from hype
| Common claim | What the evidence shows | Source |
|---|---|---|
| AI gives attackers new superweapons | Providers consistently report acceleration of existing techniques, not novel capability | OpenAI (October 2025); GTIG (November 2025) |
| Self-writing malware is everywhere | A handful of families call models at run time; Google describes the activity as nascent and PROMPTFLUX as unable to compromise a network yet | GTIG (November 2025) |
| AI phishing is more effective | Supported: 54 percent click-through against 12 percent in Microsoft's data | Microsoft Digital Defense Report 2025, as reported by The Register |
| Fully autonomous attacks are here | Partly: agents did most of the work in one documented campaign, but humans still made key decisions and hallucinations limited results | Anthropic (November 2025) |
| AI lowers the skill barrier | Supported: low-skill ransomware development and North Korean job fraud both relied on AI | Anthropic (August 2025) |
| State use is growing | Supported: Microsoft counted AI-generated content samples from government-backed groups rising from none in July 2023 to about 225 | Microsoft Digital Defense Report 2025 |
The NCSC's May 2025 assessment adds the strategic picture. It judges it almost certain that AI will make parts of intrusion operations more effective and efficient, highly likely that AI-enabled tools will spread to more state and non-state actors, and a realistic possibility that critical systems become more vulnerable to advanced actors by 2027 if mitigations lag. It notes that the time between disclosure and exploitation has already shrunk to days and that AI will almost certainly shorten it further. Its central concern is a digital divide between organisations that keep pace and the many that do not.
- Financial: more convincing payment and invoice fraud (T1657 Financial Theft) and extortion without encryption.
- Operational: faster exploitation of unpatched internet-facing systems.
- Trust: voice and video can no longer prove identity on their own.
Detection
Detecting that a model wrote something is unreliable and rarely useful. Detect the behaviour instead: the login, the payment change, the scripted command, the speed.
| Signal | Where to look | Relevant attack |
|---|---|---|
| Payment or bank detail changes requested by voice, video or chat | Finance workflow, ERP change logs | Deepfake and AI-written payment fraud |
| Login from new device right after a reported call or message, or MFA reset requests | Identity provider sign-in logs, help desk tickets | Voice-led credential theft |
| Bursts of reconnaissance and exploitation at machine speed from one source | WAF, EDR, NDR, cloud audit logs | Agentic intrusion |
| Processes calling LLM APIs or Hugging Face from servers or endpoints that should not | Proxy and DNS logs, EDR network events | Run-time AI malware such as PROMPTSTEAL |
| AI command-line tools invoked by unexpected parent processes or searching for secrets | EDR process trees, developer workstation telemetry | Credential stealers such as QUIETVAULT |
| Obfuscated PowerShell or scripts generated on the fly (T1027, T1059.001) | Script block logging, AMSI telemetry | AI-generated or rewritten payloads |
| Candidates whose interview performance, identity documents and work location do not match | HR and identity proofing records, device location | AI-assisted fraudulent hiring |
The AI SOC Guide covers using AI on the defensive side to match attacker speed in triage.
Metrics
- Phishing-resistant MFA coverage across workforce and privileged accounts.
- Time to patch internet-facing systems listed in the CISA KEV catalogue, measured in days.
- Payment changes verified out of band as a share of all supplier bank detail changes.
- Report rate versus click rate in phishing exercises that use AI-quality lures and voice pretexts.
- Machine credential exposure: count of long-lived tokens and API keys on developer endpoints and in repositories.
Prevention and hardening
Phased approach
- Close the identity gap first. Move privileged and high-risk users to phishing-resistant MFA, starting with administrators, finance and help desk staff, then the wider workforce. Pilot with one group, keep a documented fallback method for lost authenticators, and monitor sign-in failures before expanding. See the Phishing-Resistant MFA Guide.
- Remove voice and video as proof of identity. Require call-back on a known number, or a second approver in a separate channel, for payments, bank detail changes, MFA resets and access grants. The FBI recommends verifying the sender independently and never sharing MFA codes. See the Account Recovery and Help Desk Security Guide.
- Shrink the exploitation window. Prioritise KEV-listed and internet-facing vulnerabilities with tight patch deadlines, and reduce exposed services. See the Vulnerability Management Guide.
- Govern AI tool use and egress. Allowlist which hosts may reach AI APIs, log that traffic, and alert on servers and build systems calling model endpoints unexpectedly.
- Harden developer credentials. Replace long-lived tokens with short-lived, scoped credentials, scan endpoints and repositories for secrets, and restrict what AI coding tools can read and run.
- Retrain and re-test. Update awareness training to drop "look for typos" advice and rehearse voice-clone and deepfake scenarios in tabletop exercises. Roll back nothing on the identity side without a risk sign-off; most controls here are additive and can be staged by user group.
Decision guidance
| Organisation | Start with | Add when ready |
|---|---|---|
| Small business | Passkeys or security keys for email and banking; call-back rule for every payment change | Managed email security with impersonation protection |
| Mid-sized organisation | Phishing-resistant MFA for admins and finance; KEV patch deadlines; help desk verification procedure | AI API egress monitoring; deepfake tabletop exercise |
| Large enterprise or CNI | All of the above, plus detection for machine-speed activity and run-time model calls | Threat intelligence on AI misuse, red teaming with AI tooling, identity proofing for remote hires |
Where identity fits: nearly every documented case ends in a credential, a session or a machine token. AI makes the request more convincing; strong authentication, verified recovery and short-lived machine credentials make the convincing request useless. For organisations building their own agents, the Agentic AI Security Guide covers the reverse problem.
Framework mapping
| Threat | ATT&CK or ATLAS | NIST CSF 2.0 | NIST SP 800-53 Rev. 5 |
|---|---|---|---|
| Acquiring and jailbreaking models | T1588.007; AML.T0054 | ID.RA-03 | RA-3 |
| AI-written phishing and voice pretexts | T1566.002, T1566.004; AML.T0052 | PR.AT-01 | AT-2, IA-2 |
| Credential and token theft | T1078 | PR.AA-05 | IA-5 |
| Run-time AI malware and generated scripts | T1027, T1059.001 | DE.CM-01 | SI-4 |
| Agentic collection and extortion | T1119, T1657 | DE.AE-02, RS.MA-01 | IR-4 |
Lessons from incidents
- Video calls are not verification: the Arup deepfake fraud succeeded because a convincing call replaced an independent check.
- Default credentials meet agentic speed: the JADEPUFFER agentic ransomware case shows an AI agent using harvested and default credentials to destroy a production database, a familiar weakness exploited faster.
Common failure patterns
- Buying "AI detection" instead of fixing verification: tools that try to spot synthetic text or voices miss, while a call-back rule would stop the fraud.
- Training that still teaches typos and bad grammar as the main phishing tells.
- Treating every AI headline as a new threat class, diverting budget from patching and MFA, which the NCSC's assessment points to as the deciding factor.
- Unmonitored AI egress: any process can call model APIs, so run-time AI malware blends into normal developer traffic.
- Long-lived tokens on developer laptops that AI-assisted stealers can find in seconds.
Practitioner checklist
- Write a one-page position on AI-powered attacks based on the provider and NCSC reports, not vendor marketing.
- Deploy phishing-resistant MFA for administrators, finance and help desk staff first.
- Require out-of-band verification for payment changes, MFA resets and urgent requests from executives.
- Set patch deadlines in days for internet-facing and KEV-listed vulnerabilities.
- Log and restrict which systems can reach AI model APIs and alert on unexpected callers.
- Replace long-lived developer and cloud tokens with short-lived, scoped credentials.
- Update awareness training and run a voice-clone or deepfake tabletop exercise.
- Add identity checks for remote hiring that match documents, interviews and device location.
Standards and references
- NCSC: Impact of AI on cyber threat from now to 2027
- FBI IC3: Senior US Officials Continue to be Impersonated in Malicious Messaging Campaign
- Google Threat Intelligence Group: GTIG AI Threat Tracker, Advances in Threat Actor Usage of AI Tools
- OpenAI: Disrupting malicious uses of AI, October 2025
- Anthropic: Disrupting the first reported AI-orchestrated cyber espionage campaign
- Anthropic: Detecting and countering misuse of AI, August 2025
- MITRE ATT&CK: Obtain Capabilities, Artificial Intelligence
- The Register: AI makes phishing 4.5x more effective, Microsoft says
Related NHI Mgmt Group resources: Deepfake and AI Impersonation Guide · LLM Security Guide · AI SOC Guide · DDoS Protection Guide · OT Security Guide