ChatGPT-era attack automation and the new cyber threat curve

At a glance

What this is: An on-demand webinar argues that ChatGPT has made AI-assisted cybercrime easier to execute and easier to scale.

Why it matters: It matters because identity, email, and user-behaviour controls built for human-paced abuse are under pressure from faster, cheaper, and more convincing attack generation.

👉 Watch Abnormal AI's on-demand webinar on ChatGPT-era cyber threats

Context

ChatGPT has changed the economics of cybercrime by reducing the effort needed to generate convincing attack content. The core issue for identity and access teams is not novelty, but acceleration: the same social engineering and impersonation patterns now arrive faster, at higher volume, and with less manual attacker skill.

For IAM, NHI, and human identity programmes, that shifts the centre of gravity toward detection, verification, and containment. Human users still face phishing and impersonation, but the speed and scale of AI-assisted campaigns also raise the pressure on automated controls, mailbox protection, and identity assurance workflows.

Key questions

Q: How should security teams respond to AI-generated phishing at scale?

A: Security teams should treat AI-generated phishing as a scale problem, not just a quality problem. Strengthen identity verification for risky requests, correlate campaigns across email and identity telemetry, and rehearse rapid containment for mailbox compromise or account takeover. The goal is to reduce how far a successful lure can travel before it is stopped.

Q: Why do ChatGPT-assisted attacks matter to IAM teams?

A: They matter because they compress the attacker’s time and skill requirements, which increases the number of credible attempts against users and helpdesks. IAM teams should assume more social engineering pressure on password reset, delegation, and approval workflows, then make those steps harder to abuse under repeated, fast-moving campaigns.

Q: What do organisations get wrong about AI-generated threats?

A: They often focus on whether the content looks obviously malicious, when the better question is whether the campaign is adapting faster than human review cycles can handle. AI-assisted abuse succeeds when it creates enough believable variation to bypass pattern-based awareness and trigger rushed identity decisions.

Q: How can teams reduce the impact of AI-driven impersonation attempts?

A: Teams should combine user verification, conditional access, and response playbooks that isolate suspicious activity quickly. Once impersonation reaches credential capture or account access, the most effective control is the speed of containment, not just the quality of the initial detection.

Background and context

AI-generated phishing and impersonation at scale

Large language models make it trivial to produce grammatically polished lures, pretexting scripts, and impersonation messages that would once have taken a skilled attacker time to write and test. The model does not need to be malicious to be useful to the attacker. It only needs to reduce the effort required to produce plausible text, localised variants, and role-specific outreach. That changes the economics of social engineering more than the underlying technique. The attack still depends on trust, urgency, and identity confusion, but AI reduces the cost of iteration and personalisation.

Practical implication: tighten identity verification and anti-phishing controls where trust decisions still depend on message quality.

Why ChatGPT changes attacker throughput, not just creativity

The main security consequence is throughput. Attackers can generate more variants, more quickly, and with less skill dependence, which improves campaign testing and operational resilience. That means defenders see more attempts that resemble legitimate communication, not necessarily more sophisticated malware. AI can also help attackers translate, rephrase, and adapt content to different audiences, making global campaigns easier to run. The mechanism is scale plus consistency, not magic. Defenders should assume higher message volume, faster A/B testing by adversaries, and quicker adjustment when filters start catching patterns.

Practical implication: treat volume and variation as threat signals and tune detection for campaign acceleration, not only payload novelty.

Human-centric controls under pressure from machine-assisted abuse

Most enterprise identity controls still assume a human attacker has limited time, limited writing quality, and a relatively narrow operating window. AI-assisted abuse weakens all three assumptions. That does not make human identity controls obsolete, but it does mean they need stronger behavioural verification, stronger conditional access, and better downstream containment when a user does click or comply. In practice, the control failure is often not the initial lure alone. It is the speed with which a successful lure can become mailbox access, account takeover, or secondary internal fraud.

Practical implication: strengthen step-up verification, anomaly detection, and containment paths after suspicious user interaction.

NHI Mgmt Group analysis

AI-assisted abuse does not replace traditional identity attack paths, it accelerates them. The article’s core claim is not that ChatGPT invents a new class of compromise, but that it lowers the labour cost of phishing, impersonation, and pretexting. That matters because most identity programmes are still tuned to human attack tempo. Practitioners should read this as a throughput problem first and a content problem second.

Human identity controls now fail under a speed assumption they were never designed to hold. Verification steps, user training, and helpdesk checks all assume the attacker must invest time in each attempt. AI collapses that assumption by allowing rapid variation, translation, and persona switching. The implication is that control design must account for machine-amplified persistence, not just malicious originality.

Identity security needs campaign-aware detection, not message-only filtering. The signal is no longer just whether one lure looks suspicious. It is whether a series of lures shares intent, sequencing, and target selection patterns that indicate coordinated abuse. That pushes practitioners toward joined-up telemetry across email, identity, endpoint, and fraud surfaces.

ChatGPT makes social engineering more industrial, which raises the value of downstream containment. If an organisation cannot prevent every AI-generated lure, it must shorten the path from first suspicion to containment. That means stronger isolation, faster identity verification, and clearer response playbooks for account access anomalies.

AI-generated threat content should be treated as an identity governance issue, not only a security awareness issue. When attackers can generate polished, role-specific lures at scale, the governance question becomes whether identity assurance is resilient enough to survive repeated, low-cost attempts. Practitioners should evaluate whether their current controls still reflect human-written attack assumptions.

From our research:

• 43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases, according to The State of Secrets in AppSec.
• Organisations maintain an average of 6 distinct secrets manager instances, creating fragmentation that undermines centralised control, according to The State of Secrets in AppSec.
• For a broader view of how secret exposure and identity sprawl intersect, see The 52 NHI breaches Report for real-world breach patterns and control failures.

What this signals

AI-assisted social engineering changes the operating tempo of identity security. The control challenge is no longer just whether users recognise a bad message, but whether the organisation can absorb large numbers of believable attempts without widening exposure. With 43% of security professionals already concerned about AI systems learning and reproducing sensitive information patterns from codebases, per The State of Secrets in AppSec, the signal is clear: attackers are learning from the same information-rich environments defenders rely on.

Identity programmes should assume repeated, low-cost attack iteration. That shifts investment toward stronger verification, better telemetry correlation, and faster containment rather than awareness alone. Teams that only optimise for individual message inspection will miss the campaign-level pattern that AI makes easier to sustain.

Campaign-aware identity defence is becoming the practical baseline. Practitioners should connect mailbox telemetry, sign-in anomalies, and helpdesk workflows so suspicious activity can be correlated across the path from lure to access. That is where the governance gap will show up first.

For practitioners

• Harden identity verification for suspicious requests Require step-up verification for payment changes, mailbox delegation, password resets, and other high-risk requests when message origin or tone is unusual. Pair the process with helpdesk scripts that resist urgency and authority cues.
• Tune detection for campaign behaviour, not single-message anomalies Correlate repeated sender patterns, persona changes, language variants, and target sequences across email and identity telemetry. AI-assisted campaigns often look moderate in isolation but obvious as a cluster.
• Shorten containment paths after user interaction Predefine fast actions for suspicious clicks, token misuse, and account takeover indicators, including mailbox isolation, session revocation, and privilege review. Speed matters because AI-assisted abuse moves quickly from lure to impact.

Key takeaways

• ChatGPT does not create a new attack category so much as it industrialises familiar social engineering patterns.
• The measurable risk is attacker throughput, because faster iteration produces more believable lures and more compromise opportunities.
• Practitioners should shift from message-centric awareness to identity verification and rapid containment across the full abuse path.

Key terms

• AI-generated phishing: Phishing content created or adapted by generative AI to look persuasive, localised, or role-specific. The security issue is not only message quality but scale, because attackers can produce many variants quickly and test what gets past users and filters.
• Campaign-aware detection: A detection approach that looks for repeated intent across multiple messages, senders, targets, or identity events rather than judging one artifact in isolation. It is especially important when AI helps attackers vary wording while preserving the same abuse pattern.
• Identity verification: The process of confirming that a request, message, or action really comes from the person or account it claims to represent. In practice, it includes step-up checks, out-of-band confirmation, and controls that reduce the chance of impersonation succeeding.
• Containment playbook: A predefined response sequence for limiting damage once suspicious identity activity is detected. It typically includes isolation, session revocation, access review, and escalation paths so teams can act quickly before compromise spreads.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.

This post draws on content published by Abnormal AI: Chapter 1 of The Convergence of AI + Cybersecurity, an on-demand webinar on ChatGPT-era threats. Read the original.