Look for flat click-rate trends, low scenario diversity, poor correlation with actual incidents, and no behavioural change across repeat tests. If the same results keep appearing while incident patterns remain unchanged, the programme is probably measuring compliance behaviour instead of meaningful risk reduction.
Why This Matters for Security Teams
A phishing programme should change behaviour, reduce exposure, and improve detection quality. When metrics stay flat, the issue is often not user awareness but programme design. Security teams can mistake repeatable test outcomes for progress, even when the organisation is still vulnerable to credential theft, session hijacking, and follow-on compromise. Control thinking matters here because a phishing programme should support broader governance under NIST SP 800-53 Rev 5 Security and Privacy Controls, not operate as a standalone reporting exercise.
The most common failure is treating click rates as the primary success measure. That creates a narrow view of risk and encourages teams to optimise the test instead of improving the organisation. If awareness content, reporting workflows, MFA adoption, and incident handling are not linked to the programme, the exercise can look healthy while business exposure remains unchanged. In practice, many security teams encounter real phishing resilience gaps only after a credential theft or mailbox compromise has already occurred, rather than through intentional measurement of behavioural change.
How It Works in Practice
To judge whether a phishing programme is improving security, look beyond pass or fail results and examine whether the programme is producing different outcomes over time. A useful programme should show that users are reporting suspicious messages faster, that high-risk groups are receiving targeted coaching, and that incident teams are seeing better quality signals from those reports. It should also test varied scenarios, because repetitive templates quickly become predictable and distort the measurement.
Strong programmes usually connect test design to real threat patterns. That means varying lures by role, channel, and pretext, then comparing results against actual phishing and business email compromise incidents. Frameworks such as CISA phishing guidance and the behavioural emphasis in MITRE ATT&CK are helpful because they force teams to think about adversary techniques, not just user clicks.
- Track whether reporting rates improve, not only whether click rates decline.
- Segment results by department, privilege level, and exposure to external mail.
- Use scenario diversity so employees do not learn the test pattern.
- Compare training outcomes with mailbox compromise, account takeover, and helpdesk abuse cases.
- Check whether repeat offenders actually get better, or just get better at avoiding the test.
It also helps to measure downstream control performance. If reported phish are not triaged quickly, if suspicious messages are not removed from inboxes, or if MFA fatigue and token theft still lead to compromise, the programme is not reducing risk. These controls tend to break down in large organisations with inconsistent local ownership because awareness teams, SOC analysts, and identity teams often measure different things and never close the loop.
Common Variations and Edge Cases
Tighter phishing measurement often increases operational overhead, requiring organisations to balance user-facing realism against fairness, privacy, and programme fatigue. Not every flat metric means failure. In mature environments, stable click rates can coexist with stronger resilience if reporting behaviour, containment speed, and incident impact are improving. Current guidance suggests the programme should be judged on a bundle of indicators, not a single percentage.
Edge cases matter. Highly regulated teams may need different scenario design for finance, executives, or customer support because those groups face different attack paths. Remote and multilingual workforces can also distort results if templates are not localised or if training language is inconsistent. There is no universal standard for how often to test, but over-testing often trains people to spot simulations instead of teaching them to recognise real threats.
For a phishing programme to be credible, it must stay aligned to actual attack patterns and governance outcomes. That is where MITRE ATT&CK knowledge of enterprise techniques and NIST-style control mapping are useful: they help teams ask whether the programme is changing risk, not just producing completion records.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK and OWASP Agentic AI Top 10 address the attack and risk surface, while NIST CSF 2.0, NIST AI RMF and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.AT | Awareness and training should reduce risk, not just record attendance or clicks. |
| NIST AI RMF | Risk management principles help assess whether the programme reduces measurable exposure. | |
| MITRE ATT&CK | T1566 | Phishing techniques map directly to adversary tradecraft and test realism. |
| OWASP Agentic AI Top 10 | Useful where phishing trains users to resist prompt-based social engineering against AI assistants. | |
| NIST SP 800-53 Rev 5 | AT-2 | Security awareness training must be tuned and evidenced, not merely delivered. |
Measure whether training changes behaviour, reporting, and incident outcomes rather than only completion rates.
Related resources from NHI Mgmt Group
- What signals show that a cloud native security programme is too dependent on scanning?
- What signals show that a security programme is becoming more adaptive?
- What signals show that AI offensive testing is improving security outcomes?
- How can organisations tell whether their data security programme is actually improving?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on August 21, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
