Generative AI is making phishing harder to spot and defend

At a glance

What this is: This is an on-demand webinar about how generative AI is changing phishing and email threat patterns, with a focus on defender response.

Why it matters: It matters because email remains a primary entry point for identity compromise, and IAM, PAM, NHI, and human identity teams all depend on stronger detection and response when social engineering becomes more convincing.

By the numbers:

• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.

👉 Watch Abnormal AI's on-demand webinar on AI-generated phishing and defence

Context

Generative AI has lowered the skill barrier for phishing, making inbox-based attacks more scalable, more polished, and harder for users and filters to distinguish from legitimate mail. For identity teams, that means email can no longer be treated as a purely human-awareness problem because successful phishing now accelerates account takeover, privilege misuse, and downstream access abuse.

The webinar frames a practical security question: how should defenders adapt when AI can generate convincing malicious content faster than traditional awareness and signature-based controls can react? That question touches human identity, access governance, and the broader identity perimeter because the first compromised interaction is often the one that leads to credential theft or session compromise.

Key questions

Q: How should security teams respond to AI-generated phishing campaigns?

A: Security teams should assume the message quality will be good enough to fool users and focus on reducing what a successful click can do. That means phishing-resistant MFA, stronger mailbox recovery checks, tight privilege scopes, and rapid session revocation. If the attacker cannot convert a click into useful identity access, the campaign loses much of its value.

Q: Why do AI-generated phishing emails increase account takeover risk?

A: They increase account takeover risk because they remove many of the clues users once relied on to spot fraud, while allowing attackers to personalise messages at scale. The result is more credential capture, more session abuse, and more trusted-channel impersonation. Organisations should treat email compromise as an identity event, not just a user-awareness failure.

Q: What controls matter most when phishing becomes harder to detect?

A: The most important controls are phishing-resistant authentication, conditional access, least privilege, and fast revocation of suspicious sessions or tokens. These controls limit the damage from one successful lure and prevent a mailbox compromise from becoming a wider access incident. Identity teams need to design for containment, not only prevention.

Q: How can organisations tell whether their anti-phishing controls are actually working?

A: Look at whether suspicious sign-in attempts are blocked, whether recovery workflows require stronger verification, and whether compromised accounts can be contained before they are used for privilege escalation. If users can still reset access, approve changes, or reach sensitive systems after a phish, the control stack is too weak.

Background and context

Why generative AI improves phishing effectiveness

Generative AI helps attackers remove the common markers that used to make phishing easy to spot, such as awkward grammar, broken formatting, and repetitive phrasing. It also lets attackers rapidly tailor messages to roles, brands, and business context, which increases the chance that a recipient will trust the message and act on it. The technical change is not that AI invents a new attack class, but that it raises the fidelity and volume of social engineering while reducing attacker effort.

Practical implication: strengthen controls that validate message origin and user action instead of relying on users to notice language mistakes.

Why inbox attacks become an identity problem

Email phishing matters because the payload is usually not the message itself but the identity event that follows, such as credential capture, MFA prompt abuse, or session hijacking. Once an attacker controls a mailbox, they can reset passwords, intercept approvals, and impersonate trusted contacts inside the business. That makes email a front door to IAM failure, not just a communications issue, especially where a single compromised account can unlock shared tools, SaaS apps, or administrative workflows.

Practical implication: connect email security telemetry to IAM and help-desk controls so suspicious mailbox activity can trigger stronger verification before account recovery or reset actions.

How AI-assisted defence changes the control model

AI-powered detection and response tools can help by analysing message patterns, sender behaviour, and anomalous interaction chains at machine speed. The important point is that defensive AI must complement, not replace, identity controls such as MFA resistance, least privilege, conditional access, and rapid session revocation. In practice, the best defence against AI-generated phishing is a layered model that reduces the value of one stolen password or one clicked link.

Practical implication: pair AI-assisted email defence with identity controls that limit what an attacker can do after initial compromise.

NHI Mgmt Group analysis

Generative AI has turned phishing from a craft problem into a scale problem. The core change is not sophistication alone, but the removal of attacker friction. When novice attackers can generate polished lures in volume, the limiting factor shifts from writing skill to targeting and operational follow-through. For practitioners, that means awareness training remains necessary but no longer sufficient as a primary control.

Email compromise should be treated as an identity compromise path, not a messaging incident. The real objective is rarely the inbox itself. Attackers want credential capture, session takeover, or a trusted channel for further access, which pushes the problem squarely into IAM, account recovery, and privilege control. Teams that separate email security from identity security are already behind the attack chain.

Human judgement is too variable to be the last line of defence against AI-generated lures. Typos and obvious formatting errors used to give users a fighting chance. That signal is eroding, which means the enterprise must rely more heavily on layered verification, phishing-resistant authentication, and downstream containment. The practical conclusion is that inbox trust must be reduced, not merely educated.

Email trust debt: AI-generated phishing exploits the assumption that users can spot malicious mail by tone, language, or formatting. That assumption no longer holds consistently, so organisations have to rethink which decisions are safe to delegate to human recognition alone.

Identity controls must absorb the burden that awareness training can no longer carry. The more convincing the lure, the more valuable it becomes to make stolen credentials less useful through MFA hardening, conditional access, least privilege, and rapid revocation. For security leaders, the question is no longer whether phishing will happen, but how much access a single click can actually unlock.

From our research:

• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate, according to AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
• That visibility gap makes The 52 NHI breaches Report a useful next step for understanding how identity failures become incident paths.

What this signals

Generative AI is making email attack quality more consistent, which means organisations should assume that human discernment alone will keep degrading as a control. The programme response is to shift trust out of the inbox and into the identity layer, where authentication strength, recovery assurance, and privilege boundaries can absorb the impact of one successful lure.

Email trust debt: the more AI improves message realism, the more the enterprise accumulates hidden reliance on user judgement as a security control. That debt shows up when phishing-resistant authentication, session controls, and recovery governance were never designed to carry the whole load.

Teams should also align anti-phishing work with the broader identity programme, because mailbox compromise often becomes a path into SaaS, admin, and support workflows. If identity telemetry, access review, and incident response do not share signals, attackers can move from one compromised inbox to broader access with very little friction.

For practitioners

• Tighten mailbox-to-identity escalation paths Require stronger verification before password resets, MFA re-enrolment, help-desk changes, or privileged account recovery when mailbox behaviour looks abnormal.
• Adopt phishing-resistant authentication Prioritise MFA methods that resist prompt abuse and token replay, especially for high-value users, administrators, and support functions.
• Connect email telemetry to IAM response Feed suspicious inbox events into identity workflows so risky message patterns can trigger conditional access tightening, session review, or forced reauthentication.
• Reduce the value of a single stolen credential Apply least privilege, scoped app access, and rapid session revocation so a successful phishing click does not become broad lateral movement.

Key takeaways

• Generative AI is making phishing more convincing and more scalable, which reduces the value of the visual cues users once relied on.
• The evidence points to a control problem, not just a content problem, because inbox compromise quickly becomes credential theft and identity abuse.
• Practitioners need identity-aware containment, including phishing-resistant MFA, stricter recovery checks, and faster session revocation.

Key terms

• AI-generated phishing: Phishing content created or refined by generative AI to look more natural, targeted, and trustworthy. The practical change is scale and quality, not a new delivery channel. For defenders, it means message inspection and user training must be paired with identity controls that limit the effect of one successful lure.
• Phishing-resistant authentication: Authentication methods designed to resist common phishing techniques such as credential replay, prompt abuse, and token theft. In practice, these methods reduce the chance that a fooled user becomes a full account takeover. They are most valuable when paired with strict recovery and session control.
• Identity compromise path: A route from a successful social engineering event to usable account access, privilege misuse, or lateral movement. It is a governance view of the attack chain, showing how one inbox click can become broader identity risk. This perspective helps teams focus on containment, not just detection.

Deepen your knowledge

NHI governance, agentic AI identity, machine identity security, and identity lifecycle management are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or programme maturity, it is worth exploring.

This post draws on content published by Abnormal AI: an on-demand webinar on how generative AI is changing phishing and email threats. Read the original.