Why do phishing report volumes not always prove that awareness is working?

Why report counts can be misleading

Phishing report volume is a useful activity signal, but it is not a complete effectiveness metric. A rising count can mean the button is visible and easy to use, but it does not prove that people recognised more phishing, reacted faster, or helped the organisation detect a broader campaign. Volume can also rise after awareness nudges without any real change in judgement.

To interpret the number correctly, separate reporting activity from reporting quality. A programme can generate many reports from the same small group of vigilant employees, while most of the workforce still misses suspicious messages. That is why repeat reporter rate, time-to-report, and simulated-phish outcomes matter more than raw totals alone.

What better indicators show awareness is improving?

Better awareness metrics show behaviour change, not just participation. NIST SP 800-63 Digital Identity Guidelines is useful here because it reinforces the wider principle that assurance depends on the strength and quality of the signal, not just the presence of one signal. In reporting programmes, that means looking for faster escalation, cleaner triage, and more accurate user judgement over time.

Useful measurements include whether employees report the same kind of lure more than once, how quickly they report after receipt, and whether their reports help security teams identify a campaign early enough to contain it. Employee feedback also matters because it shows whether the message behind the programme is understood, trusted, and acted on.

In practice, the strongest programmes also track what happens after the report. If reports trigger no visible response, no feedback loop, or no improvement in simulation behaviour, high volume may reflect compliance theatre rather than real awareness. The metric should answer whether reporting habits are becoming more disciplined and more useful to defenders.

How to read the signal in context

Reporting data should be interpreted alongside the exposure pattern. A team that sees many identical reports from the same group may have created a small number of reliable reporters, not broad awareness. A team that sees moderate volume but shorter report times and better classification may be doing better than one with higher raw counts and slow, noisy submissions.

If you want a practical benchmark, compare trends within the same population over time rather than comparing one month to another in isolation. Changes in mail volume, simulation cadence, or internal campaigns can move the numbers without changing underlying awareness. The key question is whether the organisation is learning faster than attackers are adapting.

Practitioner Guidance

What to measure: Combine volume with repeat reporter rate, median time-to-report, false-positive report quality, and simulation follow-through. If volume rises but time-to-report and accuracy do not improve, treat the programme as visibility-heavy but behaviour-light.

What to verify: Check whether reports are routed into a process that actually produces feedback, triage outcomes, and trend analysis. If users never hear back, the reporting button may be gathering data without changing behaviour.

Practitioner takeaway: Raw report counts are a starting point, not proof of awareness. The better test is whether people are learning to recognise, escalate, and act sooner in ways that measurably improve response quality.

Related resources from NHI Mgmt Group

• What are the signs that phishing awareness training is not working well enough?
• Why do phishing campaigns continue to target payment services, financial firms, and SaaS/webmail providers even when overall report volumes fall?
• How do organisations prove access governance is working during audit?
• How do organizations prove AI agent controls are actually working?