TL;DR: AI-generated phishing is lowering the cost of convincing email attacks and widening the gap between AI-enabled persuasion and legacy phishing defenses, Abnormal AI says. Email and identity controls now need stronger verification at the point of action, not just suspicion about message content.
Editorial analysis by NHI Mgmt Group, based on content published by Abnormal AI: “ChatGPT Exposed: Protecting Your Organization Against the Dark Side of AI”.
Key questions
Q: Why do AI-generated phishing campaigns increase risk for public-sector agencies?
A: They reduce the visual and linguistic clues that users and filters once depended on, while fitting the language of agency work more convincingly.
Q: What breaks when phishing campaigns are generated and iterated by AI?
A: What breaks is the assumption that phishing is slow, manual, and easy to profile.
Practitioner guidance
- Harden email verification workflows Require secondary verification for payment changes, credential resets, and sensitive requests that arrive by email, even when the language appears polished and credible.
- Update phishing simulations for AI-generated lures Test users against fluent, role-specific phishing content rather than only obvious typo-ridden messages, so training reflects current attacker tradecraft.
- Review generative AI tool exposure Inventory internal and third-party AI systems that could be abused to support campaign creation, content polishing, or malicious workflow automation.
Bottom line: ChatGPT-assisted email attacks reduce the cost of producing convincing lures and make legacy phishing cues less reliable.
Glossary terms: AI-generated phishing, Human-Agent Trust Exploitation (ASI09), Behavioural email detection, Generative AI abuse
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
AI-assisted phishing is a persuasion problem, not just a message-quality problem. Once attackers can generate fluent, context-aware email at scale, the old assumption that bad phishing is easy to spot no longer holds. Security teams have to treat the attacker’s language engine as part of the delivery system, not the whole risk. The practical conclusion is that trust signals need to move beyond text quality.
A question worth separating out:
Q: Should organisations treat generative AI as an email security issue or an identity issue?
A: Both, but identity has to carry more of the load. Generative AI changes the quality of email content, yet the real defence is verifying who is making the request, whether the channel is expected, and whether the action matches policy. That makes email security, user verification, and approval workflows part of one control plane.
👉 Read our full editorial: ChatGPT-driven email attacks expose the AI-era phishing gap