Both. The message channel delivers the lure, but the hidden prompt can also influence an AI assistant or workflow that processes the content. That makes prompt abuse, message trust, and access boundaries part of the same control problem, especially where employees rely on AI to manage email.
Why AI-assisted phishing spans email security and AI governance
AI-assisted phishing should not be treated as just a mail-filtering problem. The email is the delivery vehicle, but the real control failure often sits at the point where an employee, assistant, or workflow decides whether to trust, summarise, classify, or act on the content. That means the defensive boundary includes message authenticity, user judgment, and the AI system that interprets the lure.
The practical distinction matters because the same lure can be harmless if it is only read, or dangerous if an assistant forwards it, extracts credentials, opens links, or acts on instructions embedded in the message. When AI is involved, the question becomes whether the system can be induced to treat untrusted content as a command, source, or workflow trigger. For a useful overview of the AI-side governance problem, see NIST AI Risk Management Framework and ISO/IEC 42001:2023 AI Management System Standard.
That is why organisations need to model the full path from inbox to action. Email controls reduce delivery and exposure, but ai governance controls determine whether the organisation has bounded the behaviour of tools that read mail, summarise mail, triage requests, or assist with approvals. In practice, the right question is not whether the attack looks like phishing or AI abuse, but whether the control gap is in delivery, interpretation, or execution.
Where the control boundary fails in practice
AI-assisted phishing becomes materially worse when the message content is designed to be machine-readable as well as human-persuasive. A hidden prompt, a manipulated instruction, or a crafted request can influence an assistant that has access to inboxes, calendars, documents, tickets, or other business tools. If the assistant can act on the message without strong trust boundaries, the attacker no longer needs only a human click, they need a workflow that will comply.
That is why message handling rules must be paired with explicit trust boundaries for AI tools. If an AI assistant can read external content, it should do so under restricted authority, with clear separation between parsing untrusted text and executing privileged actions. The same design concern appears in agentic systems that can fetch, summarise, or route email on a user’s behalf, where prompt abuse and privilege abuse can converge. A practical reference point for that dimension is Agentic AI Security Policy Template, which frames registration, access, oversight, and retirement as governance decisions rather than convenience choices.
This boundary also changes what “safe” user behaviour means. Training people to spot obvious phishing still matters, but if the organisation encourages users to rely on AI to process mail, then the deciding control is often whether the AI is allowed to interpret instructions, follow links, or surface sensitive data from connected systems. That is an access and authorisation problem as much as a messaging problem.
How to decide what to secure first
Start with the highest-consequence path: what the attacker can cause if the lure is successfully processed by a human and by an AI helper. If the likely outcome is credential capture, token theft, or action on a fraudulent request, then mailbox protections, phishing-resistant authentication, and tool-authorisation boundaries all matter. If the likely outcome is automated action on a malicious instruction, then AI governance and workflow constraints deserve equal priority.
The best control set is layered, not either-or. Email security reduces exposure, but AI governance should define what untrusted content may do, what tools it may reach, and what approvals remain human-only. For organisations building or buying those controls, the governance lens is clearer in NIST AI 600-1 GenAI Profile, which focuses on content provenance, testing, and risk management for generative AI, and NIST IR 8596 Cyber AI Profile, which bridges AI systems into cybersecurity governance.
What this means operationally is simple: if the organisation lets AI consume email, then email trust controls and AI trust controls become one control plane. Treating them separately leaves a gap where a message is blocked as spam but still consumed by a downstream assistant, or where a message reaches a user who is protected by training but not by workflow boundaries.
Risk and Threat Considerations
AI-assisted phishing raises two linked risks, exposure through the inbox and abuse through the assistant. Even when email controls catch the lure, a connected AI workflow can reintroduce the threat if it is allowed to parse, summarise, or act on untrusted content with excessive authority.
Failure mechanism: The attacker uses a convincing message, often with embedded instructions or a malicious link, to influence either the human recipient or an AI system that processes the mail. The failure occurs when trust in the message is stronger than the trust boundary around the tool that handles it.
Impact: The result can be credential theft, fraudulent approval, unwanted data disclosure, or delegated action carried out by an assistant that should have treated the content as untrusted input. At scale, the same weakness can turn a single phishing message into repeated abuse across every workflow that consumes inbox content.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack surface, NIST AI RMF, NIST SP 800-63 and NIST SP 800-53 Rev 5 set the technical controls, and ISO/IEC 42001:2023 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI Risk Management Framework | AI-assisted phishing depends on AI trust, misuse, and governance boundaries. |
| Recommendation — Map AI-assisted phishing workflows to AI RMF controls and bound untrusted content handling. | ||
| ISO/IEC 42001:2023 | AI Management System | The question asks how AI governance should share responsibility with email security. |
| Recommendation — Use AI management system controls to govern AI tools that process untrusted email. | ||
| NIST SP 800-63 | Digital Identity Guidelines | Phishing succeeds when authentication and user trust are bypassed through messages. |
| Recommendation — Adopt phishing-resistant authentication for accounts that receive phishing pressure. | ||
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | AI helpers handling email can be abused through delegated authority and overreach. |
| Recommendation — Constrain agent authority and require approval before privileged email-driven actions. | ||
| NIST SP 800-53 Rev 5 | IA-5 — Authenticator Management | AI-assisted phishing often aims to steal or misuse credentials and tokens. |
| AC-6 — Least Privilege | Email-processing AI should not inherit broad authority from the user’s mailbox context. | |
| Recommendation — Rotate and protect authenticators that phishing lures target. Limit AI and mailbox workflow permissions to the minimum necessary. | ||
Practitioner Guidance
What to prioritise: Protect the action path, not just the inbox. If an AI tool can read mail, verify whether it can also click, reply, forward, approve, export, or retrieve data from connected systems. Those are the behaviours that turn a lure into loss.
Decision rule: If the organisation is using AI to process email, treat untrusted email as hostile input to a workflow. Constrain the assistant’s authority, require human approval for sensitive actions, and separate summarisation from execution so that reading a message never implies trust.
What good looks like: Mail filtering, user awareness, and AI workflow permissions are aligned so that the same phishing lure cannot succeed through either a person or a connected assistant. The practical test is whether a malicious message can still trigger a privileged action after being “helped” by AI.
Practitioner takeaway: AI-assisted phishing is an overlap problem, so the control model must cover both message trust and delegated execution authority. If either side is weak, the other can be bypassed.
Related resources from NHI Mgmt Group
- Should organisations treat AI-driven exposure as a data governance issue or an identity issue?
- Should organisations treat generative AI as an email security issue or an identity issue?
- Should organisations treat search results and email as part of AI security governance?
- Should organisations treat prompt injection as an AI governance issue or an access control issue?