What should organisations measure in adaptive security awareness programmes?

Why This Matters for Security Teams

Adaptive security awareness programmes only work when they measure behaviour under pressure, not classroom attendance. For NHI Management Group, the practical issue is the same one seen in breaches such as the Microsoft Midnight Blizzard breach and the Salt Typhoon US telecoms breach: attackers exploit human decisions, weak reporting habits, and delayed response more than policy documents. That is why completion rates alone are weak evidence. They say a person viewed content, not whether they paused on a phishing lure, reported a suspicious prompt, or avoided reusing a password after coaching.

Measurement should instead track repeat risky behaviour, reporting speed, and whether high-risk groups improve after targeted interventions. This aligns with the NIST Cybersecurity Framework 2.0 focus on governance, awareness, and continuous improvement. It also reflects a larger NHI lesson: exposure changes faster than annual training cycles. In practice, many security teams discover the programme failed only after a user clicked, reported late, or repeated the same mistake during a real attack.

How It Works in Practice

Effective measurement starts by defining the risky behaviour the programme is meant to change. That usually means separating awareness metrics into observable actions and outcome metrics. For example, if a campaign targets phishing, the team should measure whether users report suspicious messages, how quickly they report, whether they enter credentials, and whether repeat clickers improve after coaching. If the goal is secret hygiene, the programme should measure whether people stop storing secrets in code or share them through unmanaged channels.

• Track repeat risky behaviour by user, team, and campaign type.
• Measure report rate, not just click rate, to show whether staff escalate suspicious activity.
• Measure time to report and time to contain to test whether awareness shortens response.
• Measure improvement after coaching for high-risk groups, not just overall averages.
• Use live simulations and production-adjacent scenarios, not only annual quizzes.

These measures are stronger because they connect awareness to operational risk. The NIST Cybersecurity Framework 2.0 encourages organisations to treat awareness as part of risk management, while NHI research shows why behaviour matters: only only 5.7% of organisations have full visibility into their service accounts, and 79% have experienced secrets leaks. That matters because human error often creates the initial foothold that later exposes non-human identities and other sensitive access paths. The best programmes therefore compare pre-intervention and post-intervention behaviour, then refine training based on the weakest patterns. These controls tend to break down when metrics are collected in isolated training tools because they miss what users do in real email, chat, and access workflows.

Common Variations and Edge Cases

Tighter measurement often increases privacy and administrative overhead, requiring organisations to balance visibility against employee trust and data-minimisation limits. That tradeoff is real, especially in regulated environments or unionised workforces.

Current guidance suggests avoiding vanity metrics such as course completion, attendance, or policy acknowledgement as primary indicators. Those can still be useful for governance reporting, but they do not prove behavioural change. A more defensible approach is to segment by role and exposure level. High-risk users such as finance, executives, IT admins, and help desk staff often need different thresholds because their attack surface is not the same. For low-volume teams, a single incident may distort percentages, so trend analysis over time is more useful than one-month snapshots.

There is no universal standard for exactly how many campaigns or simulations are enough. The right cadence depends on business seasonality, turnover, threat profile, and prior incident history. Organisations should also avoid overfitting measurements to phishing alone. In some environments, the bigger risk is credential sharing, poor device handling, or delayed escalation of suspicious MFA prompts. The most reliable programmes measure whether behaviour changes in the actual conditions where failure is most expensive.

Related resources from NHI Mgmt Group

• How should organisations measure trust across AI use cases, agents, and models?
• What do organisations get wrong about user friction in security controls?
• How can organisations tell whether identity and AI security controls are aligned?
• What should security teams measure to know if graymail controls are working?