How do I measure whether security awareness is reducing risky user behaviour?

What should you measure instead of training completion?

Measure behaviour change, not attendance. The most useful signals are the actions you are trying to reduce or improve, such as phishing-report rate, credential reuse, policy violations, and privileged access mistakes. Use pre- and post-training baselines, keep the definitions stable, and compare like with like across user groups and time periods.

The point is to separate genuine reduction in risky behaviour from simple programme participation. A completion score can rise while behaviour stays flat, so the metric set should reflect observable security outcomes and be tied to the same exposure conditions each period.

How do you build a fair baseline for behaviour change?

Start with a baseline window long enough to smooth short-term noise, then segment by role, function, and exposure level so high-risk groups are not averaged away. If a team handles more phishing attempts, privileged actions, or customer-facing requests, raw counts will mislead unless you normalise by opportunity or compare against the same exposure pattern.

Baselines work best when they are operationally boring: the definitions do not change, the collection method does not change, and the reporting cadence stays consistent. If those inputs drift, you may be measuring the measurement system rather than the behaviour.

Security Awareness and Human Risk Management Guide is useful here because it treats awareness as behaviour change, with metrics and human risk scoring rather than course completion alone.

Which metrics show whether risky behaviour is actually falling?

The strongest indicators are the ones closest to the behaviour you want to change. Phishing-report rate shows whether people are escalating suspicious messages faster, credential-reuse events show whether password habits are improving, policy violations show whether control bypass is dropping, and privileged access mistakes show whether high-impact workflows are becoming safer.

Use ratios and rates where possible, not just totals. A lower raw error count may simply reflect fewer opportunities, fewer logged events, or a smaller user population. Good measurement separates volume from exposure, then asks whether the rate of risky action has moved in the right direction.

Security Culture Measurement Guide supports this approach because it focuses on reporting behaviour, time to report, and other outcome signals that are more meaningful than training attendance.

How do you avoid bad conclusions from awareness data?

Compare results against exposure, not just against calendar time. A spike in phishing reports may reflect better awareness, but it may also reflect a larger phishing wave. Likewise, fewer policy violations may come from improved habits, or from fewer chances to make the mistake. Without context, the same metric can point in opposite directions.

Also watch for substitution effects. Users may stop clicking links but still approve unsafe requests, reuse credentials in other systems, or make privileged-access errors under pressure. A useful measurement set checks whether risk has moved, not merely whether the original failure mode has become less visible.

Top 10 Security Awareness Programme Mistakes is relevant because it highlights why click-rate thinking, blame culture, and narrow metrics can distort programme conclusions.

Risk and Threat Considerations

Awareness programmes can create a false sense of control if teams confuse training completion with reduced exposure. The real risk is that management believes human error is improving while risky behaviours remain unchanged, especially where high-volume phishing, credential abuse, or privileged mistakes are the main path to compromise.

Failure mechanism: If exposure changes across groups or periods, the programme can appear effective even when behaviour has not changed. A shift in phishing volume, role mix, or logging quality can mask the underlying risk pattern.

Impact: Teams may underinvest in stronger controls, misread residual risk, and miss the user groups that still need targeted intervention or tighter technical guardrails.

Practitioner Guidance

What to verify: Confirm that each metric maps to a specific risky behaviour and that the logging source is stable enough to compare periods. If the definition of a policy violation or privileged mistake changes midstream, reset the baseline rather than blending the data.

Decision rule: If a metric improves only when exposure drops, treat it as inconclusive until you can normalise for opportunity. If the rate improves across comparable exposure windows, you have a stronger signal that behaviour changed rather than the environment.

What practitioners underestimate: The most useful results often come from splitting the data by role and exposure, then looking for outliers. One high-risk team that does not improve can matter more than a broad average that looks healthy.

Practitioner takeaway: Measure the behaviours that create risk, keep the baseline stable, and interpret every change through exposure and role context, or you will overstate the effect of awareness training.

Related resources from NHI Mgmt Group

• Guide: Microsoft Entra ID Protection
• Security Awareness
• Risky User
• How can security teams measure whether training is reducing risky clicking behaviour over time?
• How do security and fraud teams measure whether awareness training is actually reducing social engineering risk?