TL;DR: AI is being used by cybercriminals to scale phishing, social engineering, and other attack tactics, and Knowbe4’s whitepaper argues that AI-assisted security awareness training and simulated phishing can help organisations harden the human layer. The deeper issue is that human risk programs now have to contend with faster, more adaptive deception at machine speed.
At a glance
What this is: Knowbe4’s whitepaper says AI is being used to supercharge phishing and related attacks, while AI-enabled security awareness and simulation are positioned as the countermeasure.
Why it matters: This matters because human risk programmes now have to defend against faster, more personalised deception, which changes how security teams train users, test resilience, and measure behaviour.
Context
Phishing remains a governance problem because attacker success still depends on people making fast trust decisions under pressure. AI changes the tempo and the quality of that deception by helping criminals craft more convincing messages, more varied lures, and more scalable campaigns. For IAM and security teams, the practical issue is not whether users can spot a textbook scam, but whether the programme can keep pace with adaptive manipulation.
This is also an identity issue in the broader sense of human identity risk. Security awareness, simulation, and behavioural measurement become part of how organisations manage trust boundaries across email, collaboration, and access workflows. The article frames AI as both the threat accelerator and the defensive enabler, which is a typical starting position for organisations that already run SAT programmes but have not yet rebuilt them for AI-era phishing.
Key questions
Q: How should security teams train users when phishing emails are AI-generated?
A: Train users to validate the request, not the writing style. AI-generated phishing often removes grammar errors and generic phrasing, so the reliable test is whether the request matches business context and comes through the expected channel. The strongest programmes pair simulations with a mandatory alternate-channel verification step for sensitive actions.
Q: Why does AI make human risk harder to control?
A: AI increases scale, personalisation, and speed. That means attackers can produce more convincing messages, test more variants, and target the same user repeatedly until a lure succeeds. Human-risk programmes need adaptive controls because static awareness cannot keep pace with automated persuasion.
Q: What do security teams get wrong about phishing awareness training?
A: They often treat training as a replacement for technical containment. Awareness can reduce clicks, but it does not stop every mistake, especially under pressure or when attackers use convincing workflow-based lures. Training should be measured by lower incident impact, faster reporting, and fewer successful follow-on actions.
Q: How can AI improve a human risk management programme without reducing control?
A: AI can help generate timely simulations, personalise examples, and keep training aligned to current attack patterns. Control is preserved when security teams approve content, review targeting rules, and use outcome metrics to verify that the programme is improving decisions rather than just increasing message volume.
Technical breakdown
How AI changes phishing delivery and targeting
AI lowers the cost of tailoring lures at scale. Attackers can vary language, tone, timing, and context quickly enough to test which messages trigger clicks, credential entry, or document opens. That changes phishing from a mostly static threat into an iterative optimisation problem. The defender is no longer evaluating one suspicious email template, but a campaign pattern that can mutate across channels and users. For security awareness, that means content-based detection by employees becomes less reliable unless training reflects realistic, adaptive examples.
Practical implication: treat phishing simulation as a continuously updated control, not a one-off awareness exercise.
Why human risk management now needs behavioural measurement
Human risk management goes beyond annual training because it attempts to influence observable behaviour. In this model, security teams need to understand who is susceptible, which scenarios still work, and whether training changes outcomes over time. AI matters because it can both increase attack variability and improve the precision of internal coaching and simulation. The important shift is from awareness as a checkbox to behaviour as a measurable security signal tied to risk reduction.
Practical implication: measure click, report, and credential-entry trends by scenario so interventions can be targeted rather than generic.
Where AI strengthens security awareness programmes
AI can support the defensive side of the same problem by helping teams generate relevant training, personalise simulations, and keep examples aligned to current attack methods. That is useful when threats evolve faster than manual content development. But the control still depends on governance. If the training content, targeting logic, and follow-up actions are not reviewed, AI can create scale without improving judgement. The value comes from better relevance, not from automation for its own sake.
Practical implication: use AI to improve relevance and cadence, but keep content governance and oversight under security ownership.
Threat narrative
Attacker objective: The attacker wants to increase the success rate and scale of social engineering so that more users reveal credentials or open the door to compromise.
- Entry begins with AI-assisted phishing or social engineering that makes malicious messages more convincing and more broadly targeted.
- Credential harvesting or user compromise follows when the victim trusts the lure and interacts with the message or fake workflow.
- Impact is achieved through account compromise, fraud, or broader access to internal systems and data.
Breaches seen in the wild
- Sisense breach 2024: A credential in Sisense's GitLab reportedly opened S3 buckets of customer tokens, passwords and certificates; CISA urged a full reset.
- CISA Private-CISA GitHub leak 2026: A CISA contractor's public GitHub repo exposed AWS GovCloud admin keys, Artifactory credentials and plaintext passwords for six months.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
AI-powered phishing is primarily a human trust problem, not just a content-generation problem. The article correctly frames AI as a force multiplier for deception, but the security gap sits in how organisations train, test, and measure trust decisions. That means human risk programmes need to be designed as adaptive controls, not static education. Practitioners should treat the human layer as a managed attack surface, not a compliance exercise.
Human risk management becomes materially more valuable when it is tied to observed behaviour. Awareness alone does not tell a security team whether users can resist tailored lures or report them quickly enough. AI-assisted simulation can surface who is being targeted, how messages are evolving, and where coaching actually changes outcomes. The programme value is in measurement and adjustment, not in generic completion metrics.
Identity governance now extends into the boundaries of human interaction with access workflows. Phishing is no longer only about mailbox compromise, because attackers increasingly aim at session entry, credential submission, and trust in downstream collaboration tools. That creates a bridge between human identity risk and broader IAM control design. Organisations need tighter alignment between awareness data, authentication controls, and escalation pathways.
Security teams should expect AI to raise the volume of plausible deception before it raises the sophistication of every attack. Most organisations will see more persuasive social engineering before they see fully autonomous fraud at scale. That means the first governance challenge is resilience to volume and variation, not rare edge-case tactics. The practitioner conclusion is clear: update the human-risk model now, before AI-driven phishing becomes routine operational noise.
AI should be used to improve the relevance of security awareness, not to replace judgement. The strongest use case is faster adaptation of training and simulation to current attack patterns. If the governance model is weak, however, the same automation can simply accelerate content production without improving defence. Practitioners should therefore pair AI-enabled SAT with clear ownership, review, and outcome metrics.
What this signals
AI-driven phishing is changing the operating model for human-risk teams. Security awareness can no longer be judged by annual completion or generic click-rate reduction alone. Programme owners need to understand which scenarios still work, how quickly users report suspicious activity, and whether the control is improving decision quality over time.
Human risk management will increasingly sit alongside identity controls. Phishing outcomes often become authentication outcomes when users submit credentials or approve access requests, so the boundary between awareness and IAM is narrowing. That makes MFA design, login hardening, and incident reporting part of the same defensive story.
For practitioners
- Strengthen phishing simulations with AI-generated variants Use realistic lures that vary tone, channel, and context so testing reflects how modern campaigns adapt across users and business processes.
- Measure behavioural risk, not just completion rates Track click, report, and credential-entry behaviour by scenario and user group so coaching is based on observed exposure, not training attendance.
- Align training with current attack patterns Refresh SAT content to mirror current AI-assisted phishing themes, especially executive impersonation, internal workflow abuse, and urgent payment or access prompts.
- Connect awareness outcomes to identity controls Use repeated failures in phishing exercises to review MFA resistance, login challenge design, and escalation paths for suspected compromise.
- Govern AI use in the awareness programme Set review rules for simulation content, targeting logic, and remediation messages so AI improves relevance without weakening control over training quality.
Key takeaways
- AI-powered phishing shifts the problem from obvious spam to adaptive deception that can be produced and varied at scale.
- Human risk programmes need behavioural measurement because training completion does not prove resilience against tailored lures.
- Security teams should pair AI-assisted simulations with identity controls and governance so awareness translates into lower compromise risk.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK addresses the attack and risk surface, while CIS Controls v8 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| CIS Controls v8 | CIS-14 — Security Awareness and Skills Training | The article centres on awareness training as the control response to AI-driven phishing. |
| Recommendation — Update awareness training with scenario-based phishing exercises and role-specific reinforcement. | ||
| NIST CSF 2.0 | PR.AT-01 — Awareness and Training | Human risk management maps directly to awareness and training outcomes under CSF 2.0. |
| PR.AA-05 — Access Permissions, Entitlements and Authorizations | Phishing often becomes an access problem once credentials or approvals are stolen. | |
| Recommendation — Measure whether awareness activities change user behaviour, not just whether they are completed. Reduce phishing blast radius by tightening authentication and access assurance around high-risk workflows. | ||
| MITRE ATT&CK | TA0001;TA0006 — Initial Access; Credential Access | AI-assisted phishing is primarily an initial access and credential theft pattern. |
| Recommendation — Map phishing simulations and detections to initial access and credential-harvesting behaviours. | ||
Key terms
- Human Risk Management: The practice of managing how people interact with security controls, especially under pressure, distraction, or deception. It combines training, policy, and friction management so identity systems are still usable enough that users do not bypass them in day-to-day work.
- Phishing Simulation Workflow: A phishing simulation workflow is the process used to convert a real or representative attack message into safe training content. It preserves the lure mechanics that make the message believable while removing malicious payloads, sensitive data, and operational risk before delivery to employees.
- Social Engineering: Social engineering is the use of deception, urgency, and authority to persuade a person to reveal information or take a risky action. It targets human decision-making rather than software defects, and often turns legitimate identity workflows into the attack path.
Deepen your knowledge
NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore nhimg.org for resources that connect identity governance to the broader security disciplines your programme depends on.
Published by the NHIMG editorial team on August 2, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org