Notifications
Clear all
Topic starter
27/06/2026 1:27 pm
TL;DR: Legitimate AI platforms are being manipulated to generate convincing phishing lures, malicious scripts, and automated fraud at scale, according to Abnormal AI and GPAI. That shifts the problem from content quality to identity, trust, and abuse controls across human, machine, and agentic workflows.
NHIMG editorial — here’s why we think this discussion matters
Questions worth separating out
Q: How should security teams govern AI tools that can be abused for phishing and fraud?
A: Security teams should govern AI tools as identity-enabled abuse channels, not only as content generators.
Q: Why do legitimate AI platforms increase the success of phishing campaigns?
A: Legitimate platforms give attackers credibility, scale, and speed.
Practitioner guidance
- Review AI platform access paths for abuse potential Inventory which users, service accounts, and integrations can generate or export content from AI platforms, then flag the sessions that can be turned into phishing or fraud workflows.
- Separate legitimate content creation from high-risk automation Require extra review for prompts, connectors, and automations that can produce scripts, external messages, or bulk output, because those are the paths most likely to be weaponized.
- Add anomaly detection around AI tool usage Look for unusual prompt patterns, burst activity, repeated export behavior, and access from identities that do not normally use generative tools at volume.
What to expect at the briefing
Abnormal AI's full webinar covers the operational detail this post intentionally leaves for the source:
- Real-world examples of how legitimate AI-powered tools are being abused for fraud and phishing.
- How attackers manipulate traditional AI chatbots to generate harmful outputs.
- The defenders' perspective on adversarial AI and how teams can adapt faster.
- Practical steps for protecting organizations in an increasingly AI-enabled threat landscape.
👉 Watch Abnormal AI's on-demand webinar on adversarial AI abuse and phishing →
Adversarial AI abuse: what it means for IAM and fraud teams?
Explore further
27/06/2026 2:48 pm
Adversarial AI is a governance problem before it is a content problem. The article shows that the real issue is not whether an AI model can be made to say harmful things. The problem is that legitimate access paths can be used to industrialize phishing and fraud faster than conventional review processes can react. That means identity, entitlement, and workflow governance now sit on the front line of AI abuse prevention.
A few things that frame the scale:
- 85% of organisations lack full visibility into third-party vendors connected via OAuth apps, according to The State of Non-Human Identity Security.
- Lack of credential rotation is cited as the top cause of NHI-related attacks by 45% of organisations, according to the same research.
A question worth separating out:
Q: How do organisations know if their AI governance is too weak?
A: Weak AI governance shows up when legitimate accounts can create harmful content, export it repeatedly, and feed it into other systems without raising scrutiny. If abuse can move from prompt to campaign with little friction, the organisation is treating AI as a tool feature rather than as an identity and trust boundary.
👉 Read our full editorial: Adversarial AI is reshaping phishing and fraud at scale
