Should organisations permit AI-assisted phishing simulations in security testing?

When approved simulations become a controlled exception, not a free-for-all

Organisations can permit AI-assisted phishing simulations when the use case is tightly scoped, authorised, and monitored. The key distinction is between a bounded training exercise and unrestricted content generation. If the same workflow can also be used to craft real lures, improve realism without governance, or evade internal review, the testing capability has crossed from defensive simulation into abuse-enabling tooling.

That boundary matters because phishing simulations are effective precisely when they resemble live attacks. AI can improve targeting, variation, language quality, and scale, but those benefits also reduce friction for malicious social engineering. Good policy therefore has to define who may generate simulations, what inputs are allowed, where content can be sent, and what logs prove the exercise stayed inside its approved purpose.

What controls make AI-assisted simulations defensible?

A defensible programme treats AI as a bounded test accelerator, not an open authoring assistant. Organisations should require pre-approval for campaign goals, approved personas, target populations, and delivery channels, and should separate the simulation workflow from any general-purpose model access used elsewhere in the business. The stronger the simulation resembles real phishing, the more important it becomes to document intent, constrain distribution, and retain review evidence.

Logging and oversight are not optional extras. Teams should be able to show who created the content, which prompts or templates were used, who approved the run, when the campaign was launched, and how opt-outs or safety exemptions were handled. Where the testing platform can call external tools, access mail systems, or interact with user data, those permissions should be tightly bounded and periodically reviewed.

For organisations already formalising AI governance, it helps to anchor the programme in a broader control set rather than ad hoc judgement. NIST AI Risk Management Framework is useful for structuring governance, accountability, and monitoring around AI-enabled use cases, while NIST Cybersecurity Framework 2.0 helps tie the testing activity to govern, protect, detect, and respond outcomes. If the simulation platform relies on model or service identities, OWASP Non-Human Identity Top 10 is a practical reminder that the supporting credentials and access paths need their own controls.

Where the real failure modes appear

The biggest failure mode is policy drift, where a sanctioned testing process gradually starts to look like a content factory for real-world phishing. That can happen when teams reuse simulation prompts, export convincing templates into live channels, or leave model access broad enough that employees can generate external-facing lures outside the test programme. Another common failure is weak separation between the testing environment and production communication systems, which makes misuse harder to spot and easier to deny.

There is also a governance risk in treating “training” as a blanket justification. If the exercise includes highly persuasive targeting, brand impersonation, or realistic credential-harvesting flows, the organisation should ask whether the same result can be achieved with less exposure. In some environments, NIST Privacy Framework can help teams think about message content, employee data use, and minimisation, especially where simulations depend on detailed personnel attributes or behavioural profiling.

If the organisation uses third-party AI services or hosted testing platforms, supply-chain and access-control risks also matter. A compromised vendor account, overbroad API key, or poorly segmented testing tenant can turn a simulation workflow into an incident path. For teams that want a more operationally focused lens, CIS Benchmarks support the underlying hardening discipline, and NIST SP 800-53 Rev 5 Security and Privacy Controls maps well to access control, audit logging, and configuration management expectations around the tooling.

Practitioner Guidance

What to verify: Approve AI-assisted phishing only when the organisation can separate test use from general content generation, restrict who can run campaigns, and prove that the platform logs prompts, approvals, and delivery actions. If those facts cannot be evidenced, treat the programme as immature even if the tooling is technically impressive.

Decision rule: If the same system can be used to produce convincing external phishing without control-owner oversight, do not allow open-ended access. Keep simulation content generation inside a governed workflow, and require explicit exception handling for any use case involving brand impersonation, credential capture, or external delivery.

What practitioners underestimate: The danger is not only malicious abuse, but normalisation. Once employees get used to “AI helps with phishing testing,” the organisation can lose the distinction between sanctioned simulation, red-team activity, and ordinary persuasive messaging. That is the point where misuse becomes harder to detect and easier to rationalise.

Practitioner takeaway: Permit AI-assisted phishing simulations only when governance is strong enough that realism increases awareness without materially increasing the organisation’s own social-engineering attack surface.

Related resources from NHI Mgmt Group

• Should organisations require different controls for AI-assisted security testing?
• Should organisations allow AI-assisted request generation in security testing?
• Why is single-provider AI agent governance not enough for enterprise security?
• How should security teams handle risks from AI browser extensions?