What breaks when attack simulation programs only track click rates?

Why This Matters for Security Teams

Click rates are a narrow proxy. They tell a team who interacted with a simulation, but not whether the organisation became harder to phish, faster to report, or better at stopping the same tactic twice. That distinction matters because phishing is an attack chain, not a single event. If measurement stops at the click, the programme can reward cosmetic improvement while leaving reporting paths, escalation discipline, and mailbox controls unchanged. Guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls reinforces the need to evaluate broader protective, detective, and response capabilities rather than a lone user metric.

The operational risk is that leadership sees a downward click trend and assumes exposure is falling, when the real issue may be that users are ignoring simulations, forwarding them to IT without using the reporting mechanism, or encountering phishing variants that current templates never test. In mature programmes, the question is not “did they click” but “did the control environment improve after the exercise.” In practice, many security teams encounter the limits of click-rate reporting only after a real phishing incident exposes the gap between training metrics and actual response behaviour.

How It Works in Practice

A better simulation programme treats clicks as one signal inside a wider measurement model. The goal is to connect awareness activity to observable risk reduction across people, process, and technical controls. That means pairing simulation results with reporting rates, time-to-report, repeat-offender trends, helpdesk escalations, mailbox takedown speed, and endpoint or identity detections that show whether a lure was contained. Where available, teams should also link simulation outcomes to incident data and threat patterns in the MITRE ATT&CK Enterprise Matrix, since phishing is often a delivery step for credential theft, token abuse, or post-compromise movement.

Useful programme metrics usually include:

• Click rate, but only as a baseline indicator
• Report rate and median time to report
• Repeat exposure by department, role, or region
• Conversion from simulation to verified ticket or case
• Control correlation, such as MFA prompts, mail filtering, or detection alerts

Threat-informed simulations are stronger when templates reflect live campaigns seen in the wild. Public reporting such as CISA cyber threat advisories can help align scenarios to current lures, delivery methods, and attacker tradecraft. For organisations using generative AI to produce phishing content or defensive coaching, the risk model expands further: content quality, prompt injection, and adversarial adaptation must be governed alongside human behaviour. These controls tend to break down in distributed workforces with weak ticketing discipline and no consistent reporting workflow because the organisation cannot distinguish learning from mere avoidance.

Common Variations and Edge Cases

Tighter measurement often increases programme overhead, requiring organisations to balance richer insight against analyst time, tooling maturity, and user fatigue. That tradeoff becomes visible when teams attempt to measure every behaviour and end up with dashboards that are hard to action. Best practice is evolving, but current guidance suggests keeping a small number of decision-grade metrics that map to actual control outcomes rather than reporting volume alone.

Some environments need different emphasis. In high-regulation sectors, leaders may want evidence that simulations support control assurance, auditability, and incident readiness, not just awareness scores. In technical teams, repeated simulations can be less useful than role-based scenarios that test privileged users, finance staff, or service desk workflows. For organisations experimenting with AI-generated phishing and automated response, the risk surface is broader and may intersect with adversarial AI methods described in the MITRE ATLAS adversarial AI threat matrix and recent analysis such as Anthropic — first AI-orchestrated cyber espionage campaign report. The practical lesson is that simulation metrics should evolve with attacker behaviour, not remain frozen around a single vanity indicator.

Related resources from NHI Mgmt Group

• What breaks when social engineering testing only tracks click rates?
• What breaks when observability tools only track latency, tokens, and error rates for AI agents?
• What breaks when security teams only track file access and not file lineage?
• What breaks when simulation platforms are shared across contractors and internal teams?