Should SOC and awareness teams use AI to generate phishing simulations?

What AI adds to phishing simulation work

AI can help SOC and awareness teams accelerate the safe parts of phishing simulation: drafting variants, localising language, adapting tone, and reusing known campaign patterns. That only helps when the team treats AI as a controlled drafting aid, not as an autonomous author. The practical question is whether the workflow preserves review, sanitisation, and approval before anything reaches employees.

The reason this matters is that phishing simulations are not just content exercises. They touch internal trust, employee privacy, brand risk, and training quality. If AI is allowed to improvise without guardrails, it can easily recreate real attacker language too closely, expose sensitive internal details, or produce scenarios that are persuasive for the wrong reasons.

For teams building this capability, the right mental model is governed reuse: take a real threat pattern, remove malicious payloads, redact sensitive context, then adapt it for learning outcomes. That is different from asking a model to invent a synthetic lure from scratch, because invention often drifts away from the actual threat patterns you want staff to recognise.

Where the workflow goes wrong

The biggest failure mode is over-trusting generated content. A model can produce a polished simulation that still leaks internal names, references live vendors, mimics an active incident too closely, or includes instructions that are no longer appropriate for awareness testing. That is why human review is not optional when the output will be sent to employees.

Another common mistake is using AI to shortcut governance. If teams skip the content approval step because the simulation "looks safe", they may accidentally create a privacy issue or an operational issue. The risk is not only malicious content. It is also poor-quality training material that teaches the wrong lesson, confuses reporting metrics, or undermines employee trust in the programme.

At scale, the problem becomes consistency. One AI-generated scenario may be carefully reviewed, while another is pushed out quickly because the team is under pressure to produce more simulations. That creates uneven quality and makes it harder to prove that the programme is repeatable, intentional, and aligned with policy.

What good governance looks like in practice

AI is most defensible when it is constrained by a standard workflow that starts with a known threat pattern, passes through sanitisation, and ends with explicit approval. Teams should use a policy template for governed AI use if they want a practical starting point for registration, oversight, monitoring, and retirement of the workflow itself.

That workflow should preserve the point of the exercise: realistic but safe learning. If the simulation is supposed to train users on credential theft, vendor impersonation, or social engineering, the AI should help translate an existing threat into a safe training artifact, not invent new attack logic. For example, teams can safely reuse patterns from the Mailchimp breach 2022 or the Dropbox GitHub breach 2022 as context for social engineering and secret exposure lessons, but only after removing sensitive operational details.

Good governance also means knowing when the model should stop. If the prompt would require live user identities, active incidents, confidential internal phrasing, or highly specific business references, that is a sign the scenario should be rewritten manually. AI can assist with structure and variation, but the team should own the final judgment about realism, safety, and appropriateness.

Practitioner Guidance

What to verify: Verify that every generated simulation has been stripped of malicious instructions, live secrets, and unnecessary personal or business identifiers before it reaches an employee inbox.

Decision rule: If the AI output is good enough to send without review, the workflow is probably too loose; keep human approval as the release gate for any outward-facing simulation.

Common mistake: Teams often optimise for speed and novelty, then discover that the best-looking simulations are the ones most likely to leak context or blur the line between training and real-world phishing.

Practitioner takeaway: Use AI to compress the drafting cycle, not to replace the controls that make phishing simulations safe, credible, and defensible.

Related resources from NHI Mgmt Group

• How should SOC teams use AI to keep pace with modern phishing investigations?
• How should security teams use AI in the SOC without weakening human oversight?
• How should security teams use AI in the SOC without losing human control?
• What should security teams control in AI-powered phishing simulations?