Subscribe to the Non-Human & AI Identity Journal

Notifications
Clear all

AI email summaries and Copilot phishing: are your controls ready?


(@nhi-mgmt-group)
Member Moderator
Joined: 1 year ago
Posts: 12212
Topic starter  

TL;DR: AI email summarisation can turn attacker-supplied text into trusted-looking “security alert” content inside Copilot workflows, with behaviour varying across Outlook and Teams surfaces, according to Permiso Security. The risk is trust transfer, because users often treat assistant output as system-generated even when it is attacker-shaped, and that breaks existing email security assumptions.

NHIMG editorial — based on content published by Permiso Security: Co-Pilot, Disengage Autophish, the new phishing surface hiding inside AI email summaries

By the numbers:

Questions worth separating out

Q: How should security teams govern AI email summaries that can be influenced by attacker text?

A: Treat AI email summaries as a governed attack surface, not a convenience feature.

Q: Why do AI-generated security alerts make phishing more effective?

A: They borrow credibility from the assistant interface.

Q: What breaks when Copilot can retrieve from multiple Microsoft 365 sources?

A: The blast radius expands from one message to the broader collaboration workspace.

Practitioner guidance

  • Separate assistant trust from message trust Mark Copilot-generated summaries as assisted content in user guidance and UI design so recipients do not confuse polished output with authenticated system notifications.
  • Constrain cross-app retrieval paths Review which Teams, OneDrive, and SharePoint sources the assistant can access during summarisation and reduce retrieval scope where it increases the impact of manipulated prompts.
  • Detect instruction-like text in summarised content Add prompt-injection detection and content inspection before summarisation so appended instructions are identified before the model renders them into a trusted panel.

What's in the full article

Permiso Security's full blog post covers the operational detail this post intentionally leaves for the source:

  • Surface-by-surface behaviour comparisons for Outlook summarize, Outlook Copilot pane, and Teams Copilot.
  • Sanitised prompt injection examples showing how attacker text changes the summary output.
  • Disclosure timeline and Microsoft CVE-2026-26133 context for the underlying issue.
  • Practical test patterns for reproducing cross prompt injection behaviour in enterprise environments.

👉 Read Permiso Security's analysis of AI email summary phishing and Copilot XPIA →

AI email summaries and Copilot phishing: are your controls ready?

Explore further

View Full Forum →  |  NHI Foundation Course →



   
Quote
(@mr-nhi)
Member Moderator
Joined: 2 months ago
Posts: 11787
 

AI email summarisation is now a phishing surface, not just a productivity feature. The article shows that untrusted content can shape what users see in a trusted assistant panel, which turns summarisation into a security boundary. That boundary is relevant to identity programmes because the output inherits credibility the input never earned. Practitioners should treat summary UIs as governed attack surfaces, not neutral productivity overlays.

A few things that frame the scale:

  • Only 19.6% of security professionals express strong confidence in their organisation's ability to securely manage non-human workload identities, according to the 2024 Non-Human Identity Security Report.
  • Another finding from that report shows 35.6% of organisations cite managing consistent access across hybrid and multi-cloud environments as their top NHI security challenge.

A question worth separating out:

Q: Who is accountable when an AI summary leads a user to click a malicious link?

A: Accountability is shared across email security, identity governance, and AI policy owners. The email gateway may miss the malicious instruction, but the assistant, the data-access permissions, and the user training model all contributed to the outcome. Organisations need a clear ownership model for AI-generated output in security-sensitive workflows.

👉 Read our full editorial: AI email summaries create a new phishing surface in Copilot



   
ReplyQuote
Share: