Malicious AI is scaling cybercrime faster than legacy defenses

At a glance

What this is: This webinar explains how malicious AI is being used to scale cybercrime through trusted genAI platforms and dark web GPTs, with the key finding that traditional detection is increasingly outpaced.

Why it matters: It matters because IAM, security operations, and AI governance teams now have to detect abuse patterns that may originate from legitimate accounts, sanctioned tools, or unmanaged shadow AI rather than obvious malware infrastructure.

Context

Generative AI is now part of the attacker workflow, not just the defender workflow. The problem for identity and security programmes is that malicious content can be produced through legitimate AI accounts, making abuse harder to distinguish from normal use unless governance shifts from static trust to runtime scrutiny.

The article focuses on cybercriminals using trusted AI platforms and purpose-built dark web GPTs to scale phishing, social engineering, and other malicious content. For IAM and NHI teams, the operational question is no longer whether AI can be blocked entirely, but how AI use, account behaviour, and output patterns are governed when the same tools can serve both productivity and abuse.

Key questions

Q: How should security teams govern AI in cybersecurity operations?

A: Security teams should govern AI in cybersecurity operations as a workflow control, not just a detection feature. Define where AI may summarise, prioritise, or route work, then keep approval authority, access changes, and exception handling under explicit human or policy control. This prevents convenience from quietly becoming delegated authority across the security programme.

Q: Why do trusted AI platforms create more risk than obvious malicious tools?

A: Trusted platforms create more risk because they blend malicious activity into ordinary business use. When attackers generate abuse through sanctioned services, simple allowlists, domain filters, or app inventory controls lose much of their value because the abuse is happening inside approved access paths.

Q: What are the signs that AI-generated malicious content is being abused at scale?

A: Common signs include sudden spikes in content volume, repeated template-like phrasing, unusually polished phishing or fraud language, and account activity that does not match normal user behaviour. Identity-linked telemetry is critical because the output alone often looks legitimate.

Q: Should organisations treat malicious GPTs as a separate threat from phishing automation?

A: Yes. Malicious GPTs change the economics of phishing and fraud by removing the manual effort that used to constrain lower-skill attackers. That means defenders should treat them as an attack-enabling capability in their own right, with separate hunting, detection, and governance assumptions.

Background and context

How malicious AI scales attacker tradecraft

Malicious AI lowers the cost of high-quality content generation, which means attackers can produce more variants of phishing, lure text, and social engineering material in less time. The important shift is not only volume, but consistency: models can maintain tone, grammar, and contextual relevance at a scale that manual operators could not. That makes campaigns more persistent and harder to separate from benign automation. For defenders, the challenge is that content quality is no longer a reliable signal of legitimacy.

Practical implication: tune detection for behavioural and contextual anomalies, not just message quality or obvious spelling errors.

Why trusted AI platforms create governance blind spots

When attackers use legitimate services such as ChatGPT, Gemini, or Claude, the abuse path blends into normal enterprise adoption of generative AI. That creates an identity problem as much as a content problem, because the same authenticated access that supports valid work can also support malicious generation. Governance based only on allowlists or app presence will miss abuse that happens inside approved tools. The control question becomes who is using the account, for what purpose, and whether that usage is compatible with policy.

Practical implication: bind AI access governance to user intent, usage monitoring, and policy enforcement rather than simple application approval.

Dark web GPTs and the cybercrime-as-a-service model

WormGPT and FraudGPT represent purpose-built malicious models rather than opportunistic misuse of consumer AI. That matters because it removes friction from attacker operations and gives cybercrime a more modular service model, where tooling is pre-tuned for abuse. In practice, this makes threat actor capability more repeatable and accessible to less skilled operators. The security significance is similar to other commoditised attack markets: scale and accessibility increase faster than traditional defence teams can adapt.

Practical implication: treat malicious model ecosystems as an adversary capability that should inform hunting, detection engineering, and threat intelligence priorities.

NHI Mgmt Group analysis

Malicious AI turns content generation into an attacker identity problem, not just a detection problem. When adversaries can generate convincing text through legitimate AI accounts, the control surface moves from message inspection to who is operating the account, under what authorisation, and for what purpose. That is why governance teams should stop treating AI misuse as a narrow threat-intelligence issue and start treating it as access misuse inside approved digital services.

Trusted AI platforms create an abuse channel that legacy security assumptions were never built to inspect. The old model assumes a visible boundary between sanctioned tools and hostile infrastructure. This article shows that boundary is collapsing because the same platforms can support productivity and abuse in the same session. Practitioners should interpret that as a policy and telemetry gap, not a tooling gap.

Malicious GPTs represent cybercrime commoditisation, which changes the defender's operating tempo. Purpose-built models such as WormGPT and FraudGPT reduce the skill threshold for attackers and expand campaign volume. That means security teams must increasingly optimise for speed of detection, threat hunting, and abuse containment rather than relying on human review to catch low-quality attacks.

Runtime governance is becoming the decisive control plane for generative AI abuse. Legacy controls built around application approval and perimeter filtering do not explain or constrain what an authenticated user does inside an AI session. The practical consequence is that organisations need to govern usage patterns, not just software inventory, because the harm emerges from how the model is used rather than which model is present.

Shadow AI and sanctioned AI now share the same abuse pattern, which is why discovery alone is insufficient. Even when the platform is approved, the behaviour can still be malicious, so inventorying tools is only the first layer. Teams should focus on policy enforcement, behavioural analytics, and identity-linked auditability so AI use can be assessed against purpose rather than assumed trust.

What this signals

Malicious AI is collapsing the distance between legitimate productivity tools and criminal tradecraft. For security programmes, that means the control objective is shifting from blocking AI outright to proving whether use is authorised, policy-aligned, and observable. The same identity and governance questions that apply to NHI abuse now apply to AI sessions, because the risk is in runtime behaviour.

Shadow AI and sanctioned AI now need the same governance lens. If an approved account can generate malicious content, discovery alone does not solve the problem. Teams need enforcement that follows the user, the session, and the output, not just the application name.

Purpose-built malicious models raise the baseline for every phishing and fraud programme. The practical consequence is that detection engineering has to assume higher-quality lures, faster campaign iteration, and more adaptive adversaries. Security leaders should expect AI-assisted abuse to become a standing input to threat modelling rather than a niche exception.

For practitioners

• Tighten AI usage policy enforcement Define acceptable AI use cases, prohibited content generation, and escalation rules for suspicious model activity across sanctioned platforms.
• Instrument identity-linked AI telemetry Correlate prompts, sessions, source identities, and output patterns so abnormal generation and abuse can be investigated in context.
• Hunt for malicious content patterns Create detections for phishing language, fraud scripts, impersonation cues, and repeated high-volume content generation that matches abuse workflows.
• Reassess shadow AI exposure Inventory unsanctioned AI tools and unmanaged accounts that can be used to generate malicious content outside approved controls.
• Align incident response for AI abuse Update triage playbooks so AI-generated malicious content, account misuse, and policy violations can be contained without treating them as isolated email or web events.

Key takeaways

• Malicious AI changes cybercrime economics by letting attackers produce convincing content faster and at greater scale.
• The article shows that trusted AI platforms and dark web GPTs both contribute to the problem, which makes simple app-level controls insufficient.
• Security teams need identity-linked governance, behaviour-based detection, and updated response playbooks to keep AI abuse from becoming normalised.

Key terms

• Malicious GPT: A malicious GPT is a generative AI system or prompt workflow used to help attackers produce harmful content, such as phishing lures, social engineering scripts, or malware support. The threat is not the model itself, but the way it reduces effort and increases scale for abusive activity.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• Identity-aware telemetry: Telemetry that includes identity, privilege, and session context rather than raw event data alone. In security operations, it ties actions to the subject that performed them, which makes correlation, triage, and investigation materially more reliable across cloud, SaaS, and on-prem environments.
• Runtime Governance: Runtime governance is the set of controls that verify what a system or agent is actually doing after deployment. It combines monitoring, authorization checks, and access validation so teams can detect drift, misuse, or excessive privilege in motion rather than assuming build-time policy still holds.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM or security governance programme, it is worth exploring.