Social Engineering Testing vs Phishing Simulation: Assurance Testing or Behaviour Change

Organisations that treat a monthly phishing simulation as their social engineering test, or a one-off social engineering test as their awareness programme, end up answering neither question they care about. A social engineering test is an assurance exercise: it measures whether processes such as help desk password and MFA resets, payment approvals and site access resist a determined, realistic attacker, and it ends in a scoped report with findings. A phishing simulation is part of a learning programme: it gives staff repeated, safe practice at spotting and reporting suspicious messages and produces trend metrics. This guide compares the two on purpose, scope, rules of engagement, ethics, metrics and failure modes, and gives decision criteria and a worked scenario. It is written for CISOs, security assurance leads, awareness managers and the HR and legal partners who approve both, and it contains no pretext scripts or lure templates.

Key takeaways

• Different questions, different owners. A social engineering test asks "would our controls stop a skilled attacker today?" and belongs to assurance; a phishing simulation asks "are staff getting better at reporting?" and belongs to the awareness programme.
• Help desk and MFA reset calls are test territory. Voice pretexts against identity verification are the scenario recent intrusions used, and they need a scoped, authorised engagement rather than a simulation platform campaign.
• Findings name processes, simulations count trends. A test reports "the reset process accepted unverified identity"; a simulation reports report rate and time to first report across quarters. Neither should name individuals for failing.
• Rules of engagement differ in depth, not in seriousness. Both need HR, legal and data protection sign-off; a test also needs a signed authorisation letter, stop conditions and a control team who can halt it in minutes.
• Click rate is a vanity metric for both. Clicks mostly reflect lure difficulty; report rate, report speed and whether the process held are the measures that change decisions.

The short answer

Run phishing simulations continuously as part of security awareness, and commission social engineering tests periodically (typically annually, or after a major change to help desk, identity or payment processes) to obtain independent assurance. The simulation programme tells you whether the workforce is building the habit of reporting; the test tells you whether the controls behind that workforce, especially identity verification, actually hold. If you can fund only one, ask what decision you need to make: a board question about whether an attacker could phone their way to an administrator account is answered only by a test. For programme design, see the Phishing Simulation Programme Guide; for how to commission and run a test safely, see the Social Engineering and Physical Penetration Testing Guide. This page covers the choice between them and how they fit together.

Objectives: measuring exposure versus changing behaviour

The NCSC frames penetration testing as "a method for gaining assurance in your organisation's vulnerability assessment", not as a primary method of finding problems, and that framing transfers directly to social engineering. A test exists to confirm or refute a belief such as "our service desk cannot be talked into an MFA reset", and its value lies in the finding and the fix. A simulation exists to give staff practice. NIST SP 800-50 Rev. 1, section 2.7.2, places phishing exercises under experiential learning and notes that they "offer opportunities to collect metrics", which is a learning purpose with measurement attached.

Mixing the two corrupts both. If a simulation is tuned for maximum realism to "catch" people, its difficulty swings from month to month and the trend becomes meaningless. If a test is constrained to the simulation platform's templates and the whole workforce, it never reaches the processes an attacker would target. The NCSC is blunt that "No training package, including phishing simulations, can teach users to spot every phishing attempt", which is why a separate assurance activity has to check the controls that must still work when someone is deceived.

Threat model for each option

Social engineering test

Threat modelled: a motivated, adaptive attacker who researches staff on professional networks, calls the help desk as an employee, and chains a password reset into MFA re-registration and privileged access. CISA advisory AA23-320A on Scattered Spider describes exactly this pattern: posing as IT or help desk staff by phone or SMS, persuading help desk staff to reset passwords or MFA tokens, SIM swapping and MFA fatigue. The relevant ATT&CK techniques are T1566.004 Spearphishing Voice, T1598.004 Spearphishing Voice (for information gathering), T1621 Multi-Factor Authentication Request Generation, T1078.002 Domain Accounts and T1199 Trusted Relationship where an outsourced help desk is abused.

Architecture in words: a control team (a small group of senior stakeholders who know the test is running) authorises a tester; the tester performs reconnaissance from public sources, then attempts agreed scenarios against named processes within agreed hours; every attempt is logged; the control team can stop the test at any time; findings go to process owners, not line managers.

Phishing simulation

Threat modelled: the high-volume, mostly email-borne phishing that reaches every inbox, mapped to T1566 Phishing and T1598 Phishing for Information (see the Phishing Guide).The simulation rehearses the last line of the email defence chain: the person who sees a message, recognises cues and reports it.

Architecture in words: a simulation platform sends rated messages to cohorts; the mail filter is configured to deliver them; a report button routes reports to the SOC triage queue, which recognises simulation reports and thanks the reporter; landing pages explain the cues without judgement; results feed aggregate dashboards with a retention limit.

Scope: help desk, MFA resets and other process targets

The most consequential social engineering scenarios rarely involve an inbox. The MGM Resorts breach reportedly began with a phone call to the IT help desk impersonating an employee found online, and the Co-op cyber attack involved a help desk password reset obtained by impersonation. In both cases the failed control was identity verification during account recovery, which no email simulation would have exercised.

Scope outsourced help desks explicitly. If a third party runs the service desk, it must give written consent, and its own verification script is part of what is being tested. Where the provider cannot be told, take it out of scope rather than test without authority. See the Account Recovery and Help Desk Security Guide for the controls a test should be checking against.

Rules of engagement, legal and ethics

Both activities process personal data about staff and both involve deception, so both need governance. The difference is that a test also needs explicit authority to do things that would otherwise be unlawful or a breach of policy.

Legal review: NIST SP 800-50 Rev. 1 says the legal team "should be included in the design and review of planned phishing exercises". For tests, legal counsel also confirms who can authorise each target and whether police or guarding contractors are told in advance. The Red Team Rules of Engagement Template gives a structure.

Monitoring law: in the UK, the ICO's guidance on monitoring workers says a DPIA is required before processing likely to result in high risk, that workers must be told about monitoring in clear language, that consent is usually unsuitable as a lawful basis in employment, and that covert monitoring is rarely justified. A simulation programme announced in policy fits that model. A test is closer to covert activity, which is one reason its data should be minimal, short-lived and reported at process level.

Ethics: the NCSC warns that "punishing people for clicking on emails you've sent starts to resemble entrapment" and says to "always check with your HR department before undertaking any phishing simulations". For tests, the equivalent risk is a service desk agent who believes they caused a breach. Agree a debrief that explains the exercise, credits anything they did right and makes clear the process, not the person, is the finding.

Metrics each one supports

NIST SP 800-50 Rev. 1 describes click-through and reporting measurements for phishing exercises; the NCSC advises that you "focus on how many people reported it". A 15-month study of more than 14,000 employees (Lain, Kostiainen and Capkun) found that embedded training shown after a simulated click did not make employees more resilient and could make them more susceptible, while a report button with crowdsourced detection was practical in a large organisation. That evidence favours reporting metrics.

See Security Awareness Metrics for a wider metric set.

Failure modes and controls by option

Social engineering test

• Failure: testing without authority over the target. Outsourced help desks, landlords and co-location sites need their own consent. Control: map ownership of every target before signing scope.
• Failure: a real change survives the test. A reset password or new authenticator left in place becomes a real weakness. Control: use test accounts, log every change and reverse it the same day with IAM confirmation.
• Failure: report names individuals. Control: write findings as process failures with root cause, for example "verification relied on employee ID and manager name, both found online".
• Failure: one-off test, no retest. Control: schedule a retest of each high finding after remediation.

Phishing simulation

• Failure: punitive programme. Naming, mandatory retraining as penalty or HR consequences destroy reporting. Control: a written non-punitive charter.
• Failure: click-rate vanity metric. Click rate falls because lures got easier, and the board is told staff improved. Control: rate difficulty (the NIST Phish Scale is one method) and report rate by band.
• Failure: harmful lures. Bonus, pay or health themes cause distress and complaints. Control: an exclusion list approved by HR.
• Failure: SOC blind to simulation reports. Control: route simulation reports through the same button and triage queue so the real path is exercised.

Decision criteria

• Choose a social engineering test when: the question is whether a specific process holds (help desk resets, payment changes, site entry); a regulator, auditor or board wants independent assurance; a process or outsourcing arrangement has just changed; or threat intelligence shows voice-based intrusion against your sector.
• Choose a phishing simulation when: the goal is workforce reporting behaviour over time; you need trend data for an awareness programme; or you are validating the report button to SOC path.
• Use both, linked, when: the organisation is large or regulated. Test findings feed simulation themes (if a vishing test succeeded, teach call-back verification), and simulation reporting data tells testers how quickly the SOC reacts.
• Small organisations: a simple announced simulation programme plus a narrowly scoped annual help desk and finance test, often bought as part of a penetration test, usually gives more value than a large platform with many campaigns.
• Large organisations: keep the two in different teams with a shared steering group, so that assurance findings are not softened to protect programme metrics. Financial entities in scope of threat-led testing regimes such as TIBER-EU should expect social engineering inside a wider threat intelligence led red team; see the Threat-Led Penetration Testing Guide.

Worked scenario

A 4,000-person insurer runs monthly email simulations. Its click rate has fallen from 14% to 4% over two years and the awareness lead presents this as success. The new CISO asks whether an attacker could reach an identity administrator by phone.

1. Reframe the metric. Difficulty ratings show recent campaigns were mostly least difficult; report rate on moderately difficult lures is flat at about one in five. The CISO changes the board measure to report rate by difficulty band and median time to first report.
2. Commission a test. Scope: the outsourced service desk (with the provider's written consent), the internal IT desk and accounts payable. Rules: test accounts only, no MFA enrolment left in place, no funds moved, control team of CISO, HR director, IAM lead and the provider's security manager.
3. Results. The internal desk refused two of three reset attempts; the outsourced desk reset a test account after the caller gave an employee number and manager name. Accounts payable required a call-back and held. The SOC was not alerted to the reset.
4. Remediate. Resets for privileged and high-value accounts now require video verification against the HR photo or manager approval through a separate channel; new MFA registrations after a reset raise a SOC alert.
5. Feed back. The awareness team adds a call-back verification module and a quarterly voice exercise for desk staff, run by the internal team with HR agreement. A retest six months later finds the outsourced desk holds.

Where identity fits

Most social engineering that matters ends in an identity event: a reset, a new authenticator, an OAuth consent or a session approval. That makes the identity provider's logs the best detection source for both activities. During a test, check whether password resets followed by new MFA method registration from an unfamiliar location raised an alert, and whether privileged role changes were caught. Phishing-resistant MFA (FIDO2 or PKI-based), which CISA recommends in AA23-320A, reduces what a successful pretext yields. See the Help Desk Identity Verification Policy Template.

Framework mapping

Practitioner checklist

• Write down the question each activity answers and who owns it, and keep them in separate teams or at least separate reports.
• Put help desk password and MFA reset verification, payment changes and outsourced service desks in the next social engineering test scope.
• Obtain written authorisation from whoever controls every test target, including third-party providers.
• Agree stop conditions, a reachable control team and same-day reversal of any account change.
• Publish a non-punitive charter for simulations and an exclusion list approved by HR and legal.
• Complete a DPIA for both, with retention limits for results and recordings.
• Report simulations by report rate and time to first report per difficulty band, not raw click rate.
• Write test findings as process failures with root cause, and retest high findings after remediation.
• Feed test findings into simulation and training themes, and simulation reporting data into SOC readiness reviews.

Standards and references

• NIST: SP 800-50 Rev. 1, Building a Cybersecurity and Privacy Learning Program
• NCSC: Phishing attacks, defending your organisation
• NCSC: Penetration testing
• CISA: Advisory AA23-320A, Scattered Spider
• ICO: Employment practices and data protection, monitoring workers
• European Central Bank: TIBER-EU framework
• Lain, Kostiainen and Capkun: Phishing in Organizations, Findings from a Large-Scale and Long-Term Study

Related NHI Mgmt Group resources: Social Engineering Guide, Security Awareness and Human Risk Management Guide, Vishing and Smishing Guide, Penetration Testing Guide.