Security Awareness Training vs Phishing Simulation: What Each Proves and How to Combine Them

Most organisations fund security awareness training and phishing simulation from one budget line, then judge both by a single click rate that neither was built to produce. Training is education: it gives people knowledge, procedures and a reason to act. Simulation is a behavioural test: it shows what people do with an unexpected message in their real inbox. This guide compares the two as separate instruments, sets out what the published evidence says about each, explains the failure modes that make programmes counterproductive, and gives a decision model for splitting effort and budget. It is written for awareness leads, CISOs and security operations managers who need to defend the mix to a board or an auditor.

Key takeaways

• They answer different questions. Training answers "do people know what to do?"; simulation answers "what do people actually do, and how fast do they tell us?". Report each against its own measures.
• Evidence for click reduction is weak for both. Two large field studies found annual training recency had no meaningful effect and embedded post-click training had a small or even negative effect.
• Reporting is the strongest outcome. Staff reports detected real campaigns quickly and at low operational cost, so report rate and time to report are the metrics worth steering by.
• Lure difficulty drives the numbers. Failure rates in one study ranged from under 2% to over 30% depending on the lure alone, so unrated click rates are not comparable between exercises.
• Neither replaces technical controls. Phishing-resistant MFA, filtering and domain-bound credentials carry the risk that humans cannot; training and simulation sit inside that layered design.

The short answer

Use security awareness training to meet the knowledge and procedure obligations (what phishing looks like, how to verify a payment change, how and where to report) and to give role-specific skills to high-risk groups. Use phishing simulation sparingly, with rated difficulty, to rehearse and measure reporting and to find which business processes attackers can mimic most easily. If you can only afford one thing beyond a baseline course, invest in a one-click report button, fast feedback to reporters and triage capacity: that is where the evidence of operational value is strongest. The Security Awareness and Human Risk Management Guide covers the wider programme; this page is about the choice between the two instruments.

The two instruments compared

What the evidence says

Two large, peer-reviewed field studies are the best available evidence, and both are uncomfortable for programmes built on click rates.

Annual training and embedded training (UC San Diego Health)

Ho and colleagues ran an eight-month randomised controlled experiment with roughly 19,500 employees at UC San Diego Health and ten simulated campaigns, presented at the IEEE Symposium on Security and Privacy in 2025. They found no meaningful relationship between how recently someone had completed annual training and whether they failed a simulation. Embedded training shown after a failure reduced failure by only about 1.7 percentage points on average. Engagement was poor: between 37% and 51% of training sessions lasted zero seconds and over three quarters lasted under a minute. Static training pages showed no benefit, while users who completed interactive training did somewhat better, which the authors caution may reflect self-selection. Over the eight months, 56% of users failed at least once. The authors recommend randomised evaluation with a no-training control group and greater investment in technical defences such as hardware MFA and password managers that only fill credentials on the correct domain. The study measured clicks only, at one organisation, and did not study reporting.

Warnings, embedded training and reporting (ETH Zurich)

Lain, Kostiainen and Capkun studied more than 14,000 employees over 15 months. Embedded training pages did not make employees more resilient: the group redirected to training clicked more on later simulations than the control group, and among respondents who remembered the page, 43% said it made them feel safe. The authors suggest a false sense of protection. In-mail warnings cut clicks sharply, and detailed warnings did no better than short ones. The reporting results were the strongest: staff kept reporting for the full 15 months without obvious fatigue, positive feedback increased later reporting, around 10% of reports on a campaign arrived within five minutes, and 918 reports of real phishing over five months exposed 252 large campaigns. Only about 1.5 reported emails a day needed manual handling once an automated filter triaged them.

What this means for the decision

• Training: keep it for knowledge, procedure and compliance, but do not expect annual modules to move click rates. Short, interactive, role-relevant content is the defensible format.
• Simulation: its value lies less in teaching non-clickers and more in rehearsing reporting and stress-testing processes. Post-click "teachable moment" pages should be short, respectful and evaluated, not assumed to work.
• Technical controls: both studies point to the same conclusion as NCSC guidance: no training, including simulation, teaches users to spot every phish, so controls must absorb the clicks that will happen.

Security awareness training in depth

Threat model. Training addresses the gap where staff do not know a pretext exists (MFA fatigue prompts, fake invoices, help desk impersonation, QR code lures) or do not know the correct response (call back on a known number, use the report button, never approve an unexpected MFA prompt).

Architecture in words. A learning management system or awareness platform assigns modules by role and joiner date, records completion (AT-4 Training Records), runs short assessments, and feeds HR and compliance reporting. NIST SP 800-50 Rev. 1 frames this as a Cybersecurity and Privacy Learning Program with a lifecycle of planning, development, delivery and evaluation aimed at behaviour change. Role-based tracks (AT-3, CIS Safeguard 14.9) cover finance approvers, help desk agents, administrators and developers.

Failure modes. Completion is treated as the outcome; content is generic and annual; modules are long enough that staff click through; procedures taught in training do not match how the business actually works (there is no callback number to use).

Controls that make it work.

1. Map each module to a procedure that exists and can be audited, such as the payment change verification step.
2. Keep modules short and interactive; the UC San Diego results favour interactive over static content.
3. Teach reporting as the primary action, including reporting after a mistake.
4. Target extra support at higher-risk roles rather than raising the volume for everyone, as NCSC advises.
5. Review content at least annually and record the review date (CIS Safeguard 14.1).

Phishing simulation in depth

Threat model. Simulation addresses habitual behaviour: people who know the rules but act quickly on a message that fits their work. It also tests the organisation, not just the individual: whether the report button works on every client, whether the security team responds, whether a process (such as a shared mailbox handling supplier invoices) is easy to mimic.

Architecture in words. A simulation platform sends lures through allow-listed infrastructure, tracks interactions per user, and redirects failures to a landing page. The report button posts to a triage queue that separates simulations from real reports; real reports go to the SOC. See the Phishing Report Button and User Reported Message Triage Guide for that pipeline, and the Phishing Simulation Programme Guide for campaign operations.

Difficulty rating. NIST Technical Note 2276, the Phish Scale User Guide, rates how hard an email is to detect. It counts observable cues (23 cues in five types: errors, technical indicators, visual presentation, language and content, common tactics) into bands of few (1 to 8), some (9 to 14) and many (15 or more), and scores premise alignment on five elements, giving a rating from minus 8 to 32 (weak at 10 or below, medium 11 to 17, strong 18 or above). The combination yields least, moderately or very difficult. NIST's point is that a high click rate on a very difficult email is expected, while a high click rate on a least difficult one signals a training need.

Failure modes. Gotcha culture, vanity click rates and reporting suppression, covered below, plus lures that exploit distressing or prohibited premises.

Measuring each: report rate and time to report

NCSC advises that metrics reward reporting, not just count clicks, so staff are not incentivised to stay quiet. Build the measurement set around that.

The Security Awareness Metrics guide covers calculation rules and board reporting. The key design choice for this comparison is that training is judged on adherence to procedures and simulation is judged on reporting, not that both are judged on clicks.

Common failure patterns

• Gotcha culture: naming, shaming or escalating repeat clickers to managers. NCSC warns that punishing staff for clicking simulations can create legal risk and that simulations can erode trust, making people less likely to report real mistakes.
• Click rate as a vanity metric: a falling click rate is reported as success when lures simply became easier. Without a difficulty rating, a drop from 20% to 5% may mean nothing.
• Reporting suppression: when reports go into a void, or a reported real phish gets the reporter a lecture, reporting falls. Lain and colleagues found positive feedback increased later reporting.
• Assuming the teachable moment teaches: landing pages that most users close in seconds, and that may create a false sense of protection, are counted as training delivered.
• Compliance theatre: annual training completion at 100% is presented as risk reduction, while payment verification is not actually performed.
• Allow-listing that weakens defences: broad filtering bypass rules created for the simulation vendor are left in place or scoped too widely; scope them to the platform's documented sending infrastructure and review them each campaign.

Ethics of lures

Simulation uses deception on your own staff, so it needs rules agreed before the first campaign.

• HR and legal sign-off: NCSC advises checking with HR before running simulations. Agree in writing what data is collected, who sees individual results and how long they are kept.
• Prohibited premises: exclude lures that promise pay rises, bonuses, redundancy news, health results, bereavement or safety alerts. They exploit distress and damage trust in genuine messages.
• Proportion: use realistic workplace processes, but do not impersonate a named colleague or a real external partner without their agreement.
• Consequences: individual results feed support, not discipline. Reserve escalation for wilful policy breaches, not for being fooled.
• Transparency: tell staff that simulations happen and why, without announcing dates. Publish aggregated results and what changed as a result.

Decision guidance: how to split the effort

For buying either capability, the Security Awareness Training Buyer's Guide lists evaluation criteria.

Designing a combined programme

1. Baseline the reporting path: deploy a one-click report button on every mail client, route reports to triage, and send each reporter feedback. Test it end to end before any simulation.
2. Write the rules of engagement: HR and legal approval, prohibited premises, data retention, and a statement that results are used for support.
3. Train for procedures: short joiner and annual modules on recognition, verification and reporting; role tracks for finance, help desk, administrators and executives.
4. Pilot simulation: one rated, least difficult campaign to a volunteer department. Check deliverability, help desk load and report triage. Roll back by pausing campaigns and removing allow-list rules if triage is overwhelmed.
5. Expand with ratings: rate every lure with the Phish Scale and mix difficulty levels. Report clicks only alongside the rating.
6. Evaluate honestly: where scale allows, hold out a randomly selected control group, as the UC San Diego authors recommend, to see whether training content changes anything.
7. Feed results into controls: processes that very difficult lures exploit get a process or technical fix, not more training.

Worked scenario: the 4% click rate

A 3,000-person logistics firm reports a falling simulation click rate, from 14% to 4% over a year, and asks whether to cut training spend. Review shows that the last four lures were generic parcel notices rated least difficult on the Phish Scale, while earlier lures mimicked the firm's own timesheet process and rated very difficult. Report rate was flat at 9%, and median time to first report was 40 minutes. Finance had approved two fraudulent supplier bank changes during the year without a callback.

The decision: the click trend is an artefact of easier lures, so it cannot justify cutting anything. The firm keeps a short annual module, replaces generic content with a payment verification module for 60 finance and procurement staff, and moves simulation budget into report button feedback and triage automation. Targets for the next year are report rate above 20% on rated campaigns, time to first report under 10 minutes, and 100% callback evidence on sampled supplier bank changes. Clicks are still tracked, but only by rating band.

Where identity fits

Most simulated lures target credentials and MFA approvals, which is why the evidence keeps pointing back to identity controls. Phishing-resistant authenticators and domain-bound credential managers remove the payoff from the lures that training struggles to stop. Phishing also reaches beyond human accounts: consent lures that grant OAuth access to applications create non-human access that persists after a password reset. See the Passwordless vs Phishing-Resistant MFA comparison for the authentication side.

Framework mapping

Practitioner checklist

• Report training and simulation against separate measures: procedure adherence for training, report rate and time to report for simulation.
• Confirm the report button works on every mail client and that every reporter gets feedback.
• Rate every simulated lure with the NIST Phish Scale and never publish an unrated click rate.
• Get HR and legal approval for rules of engagement, including prohibited premises and data retention.
• Remove any disciplinary consequence for being fooled by a simulation.
• Replace long annual modules with short, interactive, role-relevant content and record the annual review date.
• Hold out a control group when evaluating new training content, where numbers allow.
• Scope simulation allow-list rules tightly and review them every campaign.
• Route very difficult lure results to process owners and control owners, not just the awareness team.

Standards and references

• NIST: SP 800-50 Rev. 1, Building a Cybersecurity and Privacy Learning Program
• NIST: TN 2276, NIST Phish Scale User Guide
• NCSC: Phishing attacks, defending your organisation
• CIS: Control 14, Security Awareness and Skills Training
• NIST: SP 800-53 Rev. 5, Security and Privacy Controls (AT family)
• Ho et al., IEEE S&P 2025: Understanding the Efficacy of Phishing Training in Practice
• Lain, Kostiainen and Capkun: Phishing in Organizations, Findings from a Large-Scale and Long-Term Study

Related NHI Mgmt Group resources: Phishing Simulation Programme Guide · Security Awareness and Human Risk Management Guide · Phishing Report Button and User Reported Message Triage Guide · Security Awareness Metrics · Social Engineering Testing vs Phishing Simulation