AI-powered human risk management and the new phishing problem

At a glance

What this is: This whitepaper argues that AI is amplifying attacker tactics and that human risk management programs need AI-assisted training and simulation to keep pace.

Why it matters: It matters because identity and security teams still rely on people to spot deception, report suspicious activity, and avoid credential compromise, yet AI is making social engineering more convincing and scalable.

👉 Read Knowbe4's whitepaper on AI-powered human risk management and phishing

Context

AI-assisted social engineering is changing the pace and quality of phishing, pretexting, and other user-targeted attacks. The core governance gap is no longer only awareness, but whether human risk programmes can adapt fast enough to match attacker-generated variation and volume. For identity and security teams, that means the boundary between identity verification, access decisions, and user behaviour is becoming harder to defend with static training alone.

The whitepaper frames AI as both an attacker advantage and a defender capability. That is a fair starting point, but the operational question is whether organisations can turn awareness into measurable behaviour change, especially where phishing still leads to credential theft, account takeover, and downstream identity abuse.

Key questions

Q: How should organisations adapt security awareness training for generative AI phishing?

A: Security teams should move from static annual training to continuous, behaviour-focused reinforcement. Use short exercises, phishing simulations, reporting drills, and manager-supported reminders that train employees to verify requests through a second channel. The goal is not perfect detection of every message. It is faster hesitation, better escalation, and fewer successful credential captures.

Q: Why do AI-generated email attacks increase identity risk?

A: AI-generated email attacks increase identity risk because they make malicious requests more convincing at the exact point where people decide whether to trust, approve, or act. The danger is not the email alone but the downstream identity action it triggers, such as credential entry, MFA reset, or privileged approval.

Q: What do security teams get wrong about human risk management?

A: They often treat it as a training completion problem instead of a resilience problem. Completion rates do not show whether users can resist realistic lures or report them quickly. The programme should be judged by behavioural signals, especially in roles where a single compromised account can lead to broader access.

Q: How can teams reduce the damage when phishing succeeds?

A: Combine user training with phishing-resistant authentication, tighter help desk verification, and rapid detection of unusual account activity. If a lure succeeds, fast containment matters more than retrospective awareness. Teams should also predefine escalation paths for password resets, MFA changes, and suspicious session behaviour.

Technical breakdown

How AI changes phishing and social engineering at scale

AI lowers the cost of producing convincing, tailored messages and increases the speed at which attackers can test variations. Instead of sending one generic lure, an adversary can generate many context-aware messages, refine them quickly, and target different user groups with different language, timing, and pretexts. That matters because social engineering often succeeds through plausibility, not technical sophistication. The result is a more adaptive threat environment in which user vigilance is pressured by volume, personalisation, and rapid iteration.

Practical implication: awareness programmes need to test recognition of dynamic lures, not just static templates.

AI-assisted security awareness training and simulated phishing

AI can improve human risk management when it is used to generate realistic simulations, segment training by role, and reinforce lessons based on user behaviour. In practice, that means the programme should move from annual, generic training to continuous, scenario-based engagement that reflects current tactics. The value is not in making training feel automated. The value is in using automation to maintain relevance, measure response patterns, and target risky behaviour where it appears.

Practical implication: tie training content to observed attack patterns and measured user behaviour.

Why identity teams should care about the human layer

Phishing is not only an awareness problem. It is an identity compromise problem because successful deception often leads to credential theft, MFA fatigue, session hijacking, or fraudulent enrolment in downstream access flows. Once that happens, the issue moves from human judgement to access governance, because a compromised user can become a launch point for privilege abuse and lateral movement. Human risk programmes therefore belong in the same control conversation as IAM, phishing-resistant authentication, and access review.

Practical implication: treat human risk telemetry as an identity signal, not a standalone training metric.

Threat narrative

Attacker objective: The attacker wants to turn human trust into initial access and then use that access to compromise identities, systems, or data.

1. Entry occurs through AI-generated phishing or social engineering that is crafted to look legitimate and relevant to the target.
2. Credential access follows when a user submits secrets, approves a fraudulent prompt, or completes an attacker-controlled enrolment step.
3. Impact comes when the attacker uses the compromised identity to access systems, steal data, or expand to additional accounts and workflows.

NHI Mgmt Group analysis

AI has turned human risk management into an identity-adjacent control problem. Once phishing or pretexting succeeds, the consequence is rarely limited to user error. It becomes credential theft, session abuse, or fraudulent access enrolment, which places the problem squarely in the IAM and access governance conversation. Human training remains necessary, but it now functions as one layer in a wider identity control stack.

Human risk programmes fail when they measure participation instead of resilience. Completion rates and click-through metrics say little about whether users can resist realistic AI-generated lures. The better question is whether behaviour changes under pressure, especially in high-risk roles and high-frequency attack environments. That shifts programme ownership from awareness alone to measurable operational risk reduction.

Generative AI creates a trust amplification gap: attackers can scale believable deception faster than organisations can update static training. That is the core structural problem behind AI-enabled phishing. The named concept here is trust amplification gap, which describes the widening distance between attacker personalisation capability and defender training refresh cycles. Practitioners should treat that gap as a standing governance issue, not a temporary content problem.

AI should augment human risk management, but it should not be mistaken for a substitute for phishing-resistant identity controls. Training can reduce exposure, yet the durable answer still includes stronger authentication, tighter enrolment checks, and better detection of anomalous access behaviour. In practice, the most effective programmes connect user behaviour, identity telemetry, and response workflows into one control loop.

What this signals

AI-enabled phishing will push more organisations to treat human risk telemetry as part of identity governance rather than a separate awareness function. That shift matters because the control objective is no longer only to educate users. It is to detect repeated susceptibility patterns and connect them to step-up checks, enrolment safeguards, and access review workflows.

The most useful programmes will move toward continuous measurement of how users behave under realistic deception, especially in high-risk roles. Static awareness content will matter less than whether security teams can close the loop between simulation, reporting, and identity controls before attackers turn one successful lure into account compromise.

Trust amplification gap: this is the widening mismatch between attacker personalisation speed and defender training refresh cycles. Organisations that do not narrow that gap will keep relying on people to absorb an increasingly automated threat stream, which is a poor long-term control model.

For practitioners

• Build AI-aware phishing simulations Use simulations that reflect current lure styles, role-specific context, and the kinds of messages employees now encounter in email, chat, and collaboration tools.
• Track behaviour-based risk signals Measure report rates, time to report, repeat susceptibility, and risky interactions with lures so the programme reflects resilience rather than attendance.
• Connect human risk data to IAM controls Feed repeated susceptibility and suspicious interaction patterns into access review, step-up authentication, and conditional access decisions where those controls are available.
• Harden identity verification around enrolment Review password reset, MFA re-enrolment, and help desk identity checks because these are common paths from user deception to account takeover.

Key takeaways

• AI is making phishing more adaptive, which means human risk management now has to be measured as an operational control, not a communications exercise.
• The most useful programme signals are behavioural, including report rates and repeat susceptibility, because completion alone does not show resilience.
• The durable response combines AI-aware training with stronger identity controls, especially around authentication, enrolment, and account recovery.

Key terms

• Human Risk Management: The practice of managing how people interact with security controls, especially under pressure, distraction, or deception. It combines training, policy, and friction management so identity systems are still usable enough that users do not bypass them in day-to-day work.
• AI-assisted phishing: AI-assisted phishing is social engineering where generative models help create more convincing, tailored, or higher-volume lure content. The risk is not only better wording, but faster iteration, which lets attackers adapt messages until they evade filters or persuade a target to act.
• Phishing-Resistant Authentication: Phishing-resistant authentication proves identity without relying on a user to approve a prompt or reveal a reusable secret. It typically binds access to a device, key, or cryptographic proof that an attacker cannot easily reuse or coerce. This approach reduces reliance on human judgment at login time.

What's in the full article

Knowbe4's full whitepaper covers the operational detail this post intentionally leaves for the source:

• Examples of AI-assisted attacker tactics that can be used to design more realistic simulations.
• A closer look at how AI can support security awareness training and phishing exercises at scale.
• Discussion of how generative AI can be used to reinforce security culture and user engagement.

👉 Knowbe4's full whitepaper expands on attacker tactics, AI-enabled training, and simulated phishing approaches.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management in the context of real-world access risk. It helps security practitioners connect identity control design to the broader security programmes they operate.