Subscribe to the Non-Human & AI Identity Journal

Notifications
Clear all

AI phishing and deepfakes: are your email controls keeping up?


(@nhi-mgmt-group)
Member Moderator
Joined: 1 year ago
Posts: 9016
Topic starter  

TL;DR: AI now compresses target research, infrastructure build, and personalised phishing into minutes, while signature-based email security struggles against messages with no bad links, attachments, or known indicators, according to Abnormal AI. The governance problem is that controls built to detect malicious artefacts no longer match attacks that look like routine business requests and can even use real-time voice and video spoofing.

NHIMG editorial — based on content published by Abnormal AI: AI phishing and deepfake-enabled social engineering analysis

By the numbers:

  • One team cut phishing triage from 20 to 40 hours per week to 4 to 5 hours after deploying behavioral AI on top of an existing SEG.

Questions worth separating out

Q: How should security teams defend against AI-generated phishing that has no malicious links or attachments?

A: They should stop relying on content-only detection and add behavioural controls that evaluate sender history, request context, and workflow fit.

Q: Why do real-time deepfakes make callback verification less reliable?

A: Because the attacker can imitate the trusted person in the same decision window that the victim uses to verify the request.

Q: What do organisations get wrong about phishing triage when AI is involved?

A: They often treat triage as a manual review problem instead of a control-design problem.

Practitioner guidance

  • Shift email defense toward behavioural detection Measure sender, message, and workflow anomalies rather than waiting for malicious links or attachments.
  • Harden high-risk approval paths Require independent corroboration for payment, access, and account-change requests.
  • Treat verification channels as attack surface Assume audio and video can be spoofed in real time and design escalation criteria accordingly.

What's in the full article

Abnormal AI's full analysis covers the operational detail this post intentionally leaves for the source:

  • Live demonstration details showing how the attack research, infrastructure setup, and personalised message crafting were completed in minutes.
  • The Markel incident narrative, including how an image-only email bypassed the SEG and overwhelmed manual triage.
  • Operational commentary on why existing email controls struggled to inspect technically clean but highly convincing messages.
  • Examples of how behavioural AI reduced phishing review time in a live enterprise environment.

👉 Read Abnormal AI's analysis of AI-generated phishing and deepfake risk →

AI phishing and deepfakes: are your email controls keeping up?

Explore further

View Full Forum →  |  NHI Foundation Course →



   
Quote
(@mr-nhi)
Member Moderator
Joined: 2 months ago
Posts: 8472
 

Content inspection is no longer a sufficient trust model for email security. The article shows that AI-generated phishing can arrive without malicious links, bad attachments, or known signatures, which means the traditional SEG control model is being asked to solve a problem it was not built for. That is not a tuning issue. It is a control-mismatch issue. Practitioners should treat this as evidence that trust decisions are shifting from content artefacts to behavioural legitimacy.

A few things that frame the scale:

  • 70% of organisations grant AI systems more access than they would give a human employee performing the exact same job, according to the 2026 Infrastructure Identity Survey.
  • Only 44% of organisations have implemented any policies to manage their AI agents, despite 92% agreeing that governing AI agents is critical to enterprise security, according to the 2026 Infrastructure Identity Survey.

A question worth separating out:

Q: How do teams know whether their email security controls are keeping up with AI phishing?

A: Look for declining manual triage time, lower reliance on message signatures, and more accurate detection of anomalous sender behaviour. If the team still depends on suspicious links or human callback checks as the main defence, the control model is lagging behind the attack model.

👉 Read our full editorial: AI phishing now bypasses signature-based email defenses



   
ReplyQuote
Share: