Phishing Simulation Programme Guide

Most phishing simulation programmes measure the wrong thing, punish the wrong people and teach staff to distrust the security team they are supposed to report to. Done well, a simulation shows how quickly real phishing would be reported and contained; done badly, it produces a click-rate league table, complaints to HR and a workforce that hides mistakes. This guide covers programme design, ethics and consultation, scenario difficulty using the NIST Phish Scale, cadence, metrics that favour reporting over clicks, the report button and SOC feedback loop, simulation allowlisting and privacy of results. It is written for awareness leads, SOC managers and email administrators, and deliberately contains no lure templates or phishing kit instructions.

Key takeaways

• Report rate beats click rate. The NCSC advises that "as well as measuring how many people clicked on a phishing email, focus on how many people reported it", because one fast report protects everyone else.
• Rate difficulty before comparing results. The NIST Phish Scale (NIST TN 2276) combines visible cues and premise alignment into difficulty levels, so a harder email is not misread as worse behaviour.
• Clicks mostly reflect the lure. A 2025 study of more than 19,500 UC San Diego Health employees saw failure rates from 1.8% to 30.8% depending on the lure, and no significant link between recent annual training and failing.
• Punishment destroys the signal. NIST SP 800-50 Rev. 1 says simulations "should not be punitive, nor should any employee be called out for their response"; the NCSC says punishing clicks on emails you sent "starts to resemble entrapment".
• Results are monitoring data. Consult HR, legal, the data protection officer and any works council or union before launch, and give individual results access limits and a retention period.

What a simulation programme is for

A simulation programme has three legitimate goals: safe practice at recognising and reporting suspicious messages, an end-to-end test of the detection and response chain, and evidence of where technical controls leave people exposed. It is not a way to catch individuals out. The NCSC is explicit that "No training package, including phishing simulations, can teach users to spot every phishing attempt." For the wider programme, see the Security Awareness and Human Risk Management Guide; for the attack itself, see the Phishing Guide.

• Training on failure has limited effect: an ETH Zurich study of more than 14,000 employees over 15 months found embedded training after a simulated phish "does not make employees more resilient to phishing" and had side effects that could make them "even more susceptible". The UC San Diego Health study measured an average 1.7 percentage point benefit, and 37% to 51% of training sessions showed no engagement.
• Collective reporting works: the ETH Zurich study found crowdsourced employee reporting "practical in large organizations", with fast detection of new campaigns and acceptable operational load.
• Everyone fails eventually: 56% of UC San Diego Health users failed at least once in eight months, and the number rose steadily. Cumulative "ever clicked" figures say little.

Design the programme around reporting and response, keep post-click training short, and spend the effort saved on technical controls such as phishing-resistant MFA and link protection. See the Email Security Guide.

Governance, ethics and consultation

Agree the rules of engagement in writing before the first campaign. NIST SP 800-50 Rev. 1 (section 2.7.2) says the legal team "should be included in the design and review of planned phishing exercises", for example to avoid using legitimate brands, and that staff should be told exercises run on a random basis and that results guide future learning. The NCSC says you "should always check with your HR department before undertaking any phishing simulations."

• Works councils: in Germany, section 87(1) item 6 of the Works Constitution Act gives works councils co-determination over "the introduction and use of technical devices designed to monitor the behaviour or performance of the employees". A platform recording who clicked and who reported is likely to need a works agreement. Check local consultation duties elsewhere.
• Non-punitive charter: state that a simulated click or credential entry never leads to discipline, bonus reduction or naming. Deliberate, repeated policy breaches go through the ordinary disciplinary process, kept separate.
• Executives included: exemptions signal the programme is done to junior staff. NIST notes exercises can target roles "such as leadership or known administrators".

Scenario selection and difficulty

Base scenarios on what the SOC actually sees in reported and blocked mail, not on the vendor's most clickable template, and rate each one with the NIST Phish Scale before sending.

• Cues: count observable cues across five types (errors, technical indicators, visual presentation indicators, language and content, common tactics; 23 cues in all). 1 to 8 is Few (harder), 9 to 14 Some, 15 or more Many.
• Premise alignment: score five elements at 0, 2, 4, 6 or 8: mimics a workplace process; has workplace relevance; aligns with other situations or events; engenders concern over consequences for not clicking; has been the subject of targeted training or warnings. Sum the first four and subtract the fifth: 10 or below is Weak, 11 to 17 Medium, 18 or above Strong. NIST says someone "with knowledge of the target audience's work culture" should do this.

NIST warns that "click rates and reporting rates do not provide a complete picture of an organization's phishing risk"; a Least difficult email with a high click rate is the result that suggests training needs to change. The Phish Scale is free for academic use but commercial use needs a NIST licence, so ask whether your vendor's built-in rating is licensed.

What counts as a compromise

Define compromise per scenario before launch. Microsoft Defender attack simulation training is a documented model:

• Realistic premises are legitimate: attackers copy workplace tools; the Dropbox GitHub breach began with a fake CircleCI login. Use strong-premise scenarios sparingly and within the harm limits below.
• Beyond email: the Twilio breach started with SMS phishing, and ATT&CK lists T1566.004 Spearphishing Voice. Voice and SMS simulations need extra legal review; see the Vishing and Smishing Guide.

Avoiding harm: lures that are off limits

In December 2020 GoDaddy sent employees a fake email promising a $650 holiday bonus; about 500 were told they had failed and had to complete training. The company later said: "We understand some employees were upset by the phishing attempt and felt it was insensitive, for which we have apologized." Keep a written exclusion list reviewed by HR and legal:

• Pay and benefits: bonuses, pay rises, pensions, redundancy, restructuring.
• Health and welfare: pandemics, test results, bereavement, employee assistance.
• Real crises: disasters, conflicts and incidents affecting staff or families.
• Brands and public bodies: real companies, tax authorities, regulators and police, unless legal approves.
• Security's own channels: lures imitating security warnings or the report button teach people to ignore real ones.

Landing pages should explain the exercise without judgement, show the cues present and point to the report button. Never store entered passwords; confirm this in writing from any vendor.

Cadence and targeting

Neither NIST SP 800-50 Rev. 1 nor the NCSC sets a frequency, so cadence is a design choice between enough data to see trends and enough restraint that every legitimate email does not look like a test.

• Small organisations: use your email platform's built-in simulation and reporting, run quarterly, and put most effort into acknowledging reports.
• Large organisations: sample by role and region, translate scenarios, avoid local holidays, and report by business unit, never by named individual.
• Repeat clickers: respond with support and technical guardrails (stricter link handling, sandboxing, removing unneeded privileges), not sanctions. Microsoft's default repeat offender threshold is two consecutive compromises.

Measuring the right things

The NCSC warns that "if you appear to value the absence of reports of problems, you incentivise people to keep quiet about issues." Lead with reporting metrics, normalise every rate by Phish Scale difficulty and lure, and publish definitions.

• Filter bot clicks: Microsoft lists non-Microsoft filtering services and email forwarding as causes of false clicks. Discard clicks within seconds of delivery and check user agents and source IPs.
• Ignore read and delete signals: Microsoft cautions they can be wrong when users report from the reading pane.
• Predicted versus actual: Microsoft's predicted compromise rate per simulation helps separate a hard email from a weak population.

The report button and the SOC feedback loop

The NCSC asks whether reporting is "clear, simple and quick to use" and advises that you "Quickly provide feedback on what action has been taken". Build this before the first campaign.

1. One button: use the same report control for real and simulated mail. In Microsoft Defender for Office 365, User reported settings selects the built-in Report button or a non-Microsoft add-in, and sends reports to Microsoft and my reporting mailbox, My reporting mailbox only, or Microsoft only.
2. Protect the reporting mailbox: Microsoft requires it to be a SecOps mailbox in the advanced delivery policy, warning that otherwise "a user reported message might trigger a training assignment by the phishing simulation product". Exclude it from DLP.
3. Auto-close simulations: match reported simulations automatically and thank the reporter at once. Microsoft notes simulations reported through non-Microsoft tools are not captured in its simulation reports, so test mixed-vendor setups.
4. Triage real reports fast: cluster by sender, URL and subject, purge matches from all mailboxes and block indicators.
5. Close the loop: send every reporter a result (Microsoft supports custom Phishing, Junk and No threats found emails) and thank people for false alarms too.
6. Feed scenarios: use real reported campaigns to choose the next quarter's scenarios.

See the SOC Guide for triage, and the MFA Fatigue and Bypass Guide for making unexpected MFA prompts reportable, as the Uber breach showed they must be.

Simulation infrastructure and allowlisting

Third-party simulations will be filtered, detonated or clicked by security tools unless delivery is configured. Use a narrow, documented bypass for simulation traffic only. Microsoft's advanced delivery policy (Email & collaboration, Policies & rules, Threat policies, Advanced delivery, Phishing simulation tab) is a well-documented example.

• Never allowlist shared infrastructure: Microsoft warns that adding on-premises or non-Microsoft sending IPs "effectively bypasses spam filtering for any internet sender who impersonates the domain specified". Use dedicated simulation domains and IPs.
• Direct injection: "Direct Injection emails bypass transport, so Advanced Delivery policy does not apply to them"; such platforms need mailbox write permissions instead.
• Other layers: apply the same narrow pattern to gateways, URL rewriting and browser isolation, with an owner and review date per exception.

Where identity fits: simulation and reporting platforms use API tokens, OAuth app registrations or service accounts that can read reported mail, inject messages or read the directory. Grant the narrowest scopes, vault and rotate the secrets, review consents quarterly and remove them when you change vendor. See the API Key Management Guide.

Privacy of results

Individual outcomes are personal data about workers. The ICO says employers must "select the least intrusive means", lists "monitoring that may result in financial loss (such as performance management)" as high-risk processing needing a DPIA, and says "Consent is not usually appropriate in the employment context".

• Tell people what is recorded (delivery, click, credential entry, report, training), who sees it and for how long.
• Aggregate by default: leadership sees business unit and role level only.
• Allow challenge: the ICO says workers should be able to "see and, if necessary, explain or challenge the results"; scanner false clicks make this essential.
• Retain briefly and keep results out of HR files unless a DPIA, legal advice and any works agreement support it. See the DPIA Guide: Running a Data Protection Impact Assessment.

Rolling out the programme

1. Reporting first: deploy the report button, SecOps mailbox, triage and reporter feedback, and measure real report volumes for a month.
2. Charter and consultation: agree goals, the non-punitive rule, lure exclusions, metrics and data handling; complete a DPIA where results are kept per person.
3. Announce: tell staff simulations will run at random times and that reporting is what counts.
4. Test delivery: configure allowlisting with dedicated domains and IPs, then send to a seed group of security staff to check delivery, landing pages, click filtering and report matching.
5. Pilot: run two or three rated campaigns with consenting business units and review report rate, complaints and help desk load.
6. Scale: random sampling, role scenarios and quarterly reporting by business unit.
7. Roll back: withdraw any scenario that causes distress or reduces real reporting, remove allowlist entries when vendors or infrastructure change, and revisit the DPIA when purpose or data change.

Framework mapping

Common failure patterns

• Click rate as the headline. It rewards easy lures and hides whether anyone reports.
• Punitive follow-up. Mandatory courses or manager escalation after a click; the NCSC says staff afraid for their jobs "will not report mistakes".
• Exploitative lures. Bonus, health or crisis themes, as at GoDaddy, cost more trust than they buy in data.
• Reports that go nowhere. An unmonitored mailbox, or one not set as a SecOps mailbox, so reports trigger training or vanish.
• Over-broad allowlisting. Allowlisting a shared platform's IPs opens a path for real attackers spoofing that domain.
• Email only. Ignoring SMS, voice and MFA prompts, the channels used against Cisco and Twilio in 2022.

Practitioner checklist

• Deploy one report button, a SecOps reporting mailbox and reporter feedback before the first campaign.
• Agree a written charter with HR, legal, the DPO and any works council, including a kill switch.
• Rate every scenario with the NIST Phish Scale and store difficulty and lure type with results.
• Define compromise per scenario and filter scanner clicks before reporting.
• Lead with report rate and time to first report; treat click rate as context.
• Allowlist only dedicated simulation domains and IPs, and review quarterly.
• Exclude pay, health, crisis, security-team and unapproved brand lures.
• Aggregate results for leadership, allow individuals to challenge theirs, and set retention.
• Scope, vault and rotate the tokens and app consents the platforms use.

Standards and references

• NIST: TN 2276, NIST Phish Scale User Guide
• NIST: SP 800-50 Rev. 1, Building a Cybersecurity and Privacy Learning Program
• NCSC: Phishing attacks: defending your organisation
• ICO: Data protection and monitoring workers
• Microsoft Learn: Configure the advanced delivery policy for non-Microsoft phishing simulations and SecOps mailboxes
• Microsoft Learn: User reported settings
• ETH Zurich (arXiv): Phishing in Organizations: Findings from a Large-Scale and Long-Term Study
• UC San Diego: Understanding the Efficacy of Phishing Training in Practice

Related NHI Mgmt Group resources: Security Awareness and Human Risk Management Guide · Phishing Guide · AiTM Phishing Guide · Vishing and Smishing Guide · Email Security Guide