What are the signs that generative AI is increasing exposure to phishing and sensitive data leakage?

Warning signs include more convincing phishing messages, unusual requests for confidential information, and employees pasting sensitive content into external AI tools. Organisations may also see policy bypass attempts through browser access, clipboard use, or unsanctioned web applications. These signals indicate that AI risk is no longer theoretical and should be handled as a data protection and user behaviour problem.

How Generative AI Changes Phishing and Leakage Exposure

Generative AI changes this risk by improving both the quality and the scale of deception. Phishing content can be written with fewer language errors, better context matching, and more persuasive tone, which makes social engineering harder to spot. At the same time, employees may treat AI tools as convenient workspaces and paste content they would never intentionally send outside the organisation, especially when they are trying to summarise, translate, draft, or analyse it.

The security issue is not simply that AI exists, but that it lowers the effort required to produce believable messages and lowers the friction for data exfiltration through ordinary user behaviour. A phishing message that once stood out because of poor grammar may now look routine, and a sensitive document copied into a public chatbot can leave the organisation’s control boundary even if no malicious actor is directly involved. For this reason, teams should treat GenAI exposure as both a deception problem and a data-handling problem. In practice, many security teams notice the change only after staff start using external AI tools as a shortcut for routine work rather than through deliberate policy approval.

For a governance lens on this shift, NIST’s NIST AI 600-1 Generative AI Profile is useful because it frames GenAI as a risk-managed capability rather than a harmless productivity layer.

How It Shows Up in Daily Workflows

In practice, the signs usually appear in ordinary workflows before they appear in incident records. A user may receive a phishing email that reads like a manager, supplier, or client, with specific references that feel plausible enough to bypass a quick visual check. A second pattern is data leakage through convenience behaviour: a person pastes text into an external AI tool to reword it, summarise it, or extract action items, without considering whether the content contains customer data, internal strategy, credentials, legal material, or other confidential information.

Teams should watch for three broad clusters of behaviour:

More convincing inbound messages, especially where tone, terminology, and timing match the recipient’s work context.
Requests for sensitive details that appear normal because they are wrapped in a legitimate-sounding business task.
Users moving data into browser-based AI tools, extensions, or unsanctioned web apps outside approved channels.

The mechanism matters. GenAI can remove the rough edges that once made phishing easier to spot, while also making it easier for an internal user to process or repackage data in an unapproved place. That means detection should not rely only on email filtering or only on DLP. It needs a combined view of message content, browser activity, cloud application use, and policy exceptions. If you can see repeated use of external AI tools around sensitive work, that is often a stronger indicator than a single suspicious email. The guidance breaks down where the organisation cannot observe browser activity, cannot classify the data being pasted, or cannot distinguish approved AI use from unsanctioned tool adoption.

When the Pattern Stops Being “Just Productivity”

Tighter control over GenAI usage often increases friction, so organisations have to balance employee convenience against the cost of uncontrolled disclosure. That tradeoff becomes real when the same tools that speed up drafting also create an easier path for leakage or phishing abuse. Guidance in this area is still evolving, but the practical threshold is simple: when a tool is used on confidential material, it is no longer a harmless assistant in governance terms.

There are a few edge cases worth separating. Not every polished message is malicious, because legitimate senders also use AI to improve writing. Likewise, not every external AI prompt is a breach, because some queries are low risk and generic. The useful question is whether the behaviour changes the organisation’s exposure profile. If the organisation allows copying internal material into third-party AI systems, the risk is not only immediate disclosure but also retention, reuse, and weak auditability. If employees become accustomed to AI-generated language in routine work, phishing messages that borrow that style become harder for staff to challenge.

For the attacker perspective, this overlaps with broader social engineering and credential theft patterns, including the use of increasingly plausible pretexts and the exploitation of trust in normal-looking communication. For official threat context, Anthropic’s report on an AI-orchestrated cyber espionage campaign is relevant because it shows how AI can support abuse of scale, targeting, and social engineering.

Risk and Threat Considerations

The material risk is twofold: AI-assisted phishing increases the credibility of social engineering, while unsanctioned AI use increases the chance that confidential data leaves approved control boundaries. This is a trust and exposure problem, not just a content-quality problem, because the same workflow that helps users write faster can also help an attacker blend into normal communication patterns.

Failure mechanism: Attackers use GenAI to generate convincing lures, mimic tone, and tailor requests to the target’s role or context. On the leakage side, users paste sensitive text into external tools without understanding retention, access, or reuse implications, which can create uncontrolled disclosure even without malicious intent.

Impact: Organisations can face credential compromise, unauthorised disclosure of sensitive data, weaker detection of phishing attempts, and reduced confidence in email and browser-based collaboration channels. Over time, the risk can become systemic if AI use is widespread but invisible to security teams.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

MITRE ATT&CK address the attack and risk surface, while NIST AI RMF, NIST AI 600-1, NIST CSF 2.0 and CIS Controls v8 set the governance and control requirements practitioners need to meet.

Framework	Control / Reference	Relevance
NIST AI RMF	GOVERN — Govern	GenAI phishing and leakage are governance and risk-management concerns.
Recommendation — Establish AI governance to define acceptable GenAI use and review high-risk workflows.
NIST AI 600-1	MAP — Map	Identifies GenAI uses, data flows, and exposure points that create phishing and leakage risk.
Recommendation — Map GenAI use cases and data paths to spot where sensitive content can leave control.
NIST CSF 2.0	PR.AT-1 — Awareness and Training	User judgement and reporting determine whether AI-enabled phishing and leakage are recognised.
Recommendation — Train staff to recognise AI-polished phishing and to avoid pasting sensitive data into external tools.
CIS Controls v8	CIS 13 — Data Protection	Data loss through external AI tools is a control and monitoring problem.
Recommendation — Apply data protection controls to restrict sensitive content from unsanctioned AI services.
MITRE ATT&CK	T1566 — Phishing	AI improves phishing realism and targeting within a known adversary technique.
Recommendation — Track AI-enhanced phishing attempts as T1566 activity and tune detection for social engineering.

Practitioner Guidance

What to prioritise: Treat repeated use of external AI tools on sensitive material as the stronger signal, not just the appearance of better-written phishing. That behaviour shows where policy, user habit, and exposure are converging.

What to verify: Confirm whether the organisation can distinguish sanctioned AI use from browser-based shadow use, and whether logging, web filtering, or DLP actually captures copy-and-paste behaviour. If not, the team is likely underestimating leakage risk.

Common mistake: Focusing only on email security while ignoring the browser and collaboration layer. That misses the point that GenAI exposure often shows up where people draft, translate, summarise, or reformat content.

Practitioner takeaway: The most important judgement is whether GenAI is changing user behaviour in ways that make both deception and disclosure easier to normalise; once that shift happens, awareness training alone is rarely enough.