TL;DR: Attacker-supplied text can shape Copilot email summaries into trusted-looking security-alert content, Permiso Security found. Summary surfaces need separate testing because users may treat assistant output as system-generated even when it is attacker-shaped.
Editorial analysis by NHI Mgmt Group, based on content published by Permiso Security: “CO-PILOT, DISENGAGE AUTOPHISH: The New Phishing Surface Hiding Inside AI Email Summaries”.
Key questions
Q: Why do AI-generated email summaries create a phishing risk?
A: Because they can transfer credibility from the assistant to attacker-supplied content.
Q: How should security teams evaluate Copilot summary interfaces?
A: Treat each summary interface as a separate control point and test whether it flags injected instructions, ignores them, or converts them into trusted-looking action prompts.
Q: What breaks when AI email summaries can be shaped by attacker-controlled text?
A: The control that breaks is the assumption that users will inspect the original email before acting.
Practitioner guidance
- Test each Copilot summary surface separately Validate Outlook Summarize, the Outlook Copilot pane, and Teams Copilot as independent trust boundaries.
- Red-team summary-output credibility Run phishing simulations that place low-visibility instruction text inside benign-looking mail and measure whether users trust AI-generated alerts more than the original message.
- Constrain cross-workspace retrieval Review which Microsoft 365 sources Copilot can surface during email summarisation and reduce retrieval scope where the summary does not need Teams, OneDrive, or SharePoint context.
Bottom line: AI summaries can become the phishing payload when attacker-supplied text is rendered with assistant authority.
Glossary terms: Cross Prompt Injection, Model-Mediated Phishing, Trust Transfer, Summary Surface
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
Trust transfer is now a first-class identity risk: when users act on AI-generated summaries, they are responding to a trust signal that did not exist in the original email. That shifts the security problem from message filtering to interface authority. The practitioner implication is that identity-aware controls must account for where trust is formed, not just where content originates.
A few things that frame the scale:
- The IBM/Ponemon 2025 Cost of a Data Breach Report found that phishing-initiated breaches cost an average of $4.8M each.
A question worth separating out:
Q: What should organisations do when AI summaries can draw from multiple Microsoft 365 sources?
A: Limit retrieval scope to the minimum data needed for the task and review whether internal context can be folded into externally triggered summaries. If the assistant can combine email with collaboration data, the attack surface expands from phishing into context-assisted deception.
👉 Read our full editorial: AI email summaries create a new phishing surface in Copilot