Most awareness dashboards report what is easy to count, completion and click rate, and neither tells a board whether human-enabled attacks are less likely to succeed this quarter than last. This guide is the measurement layer for an awareness or human risk programme: how to tier metrics, how to write a metric so two analysts calculate the same number, which measures are KPIs and which are KRIs with thresholds, where the data comes from and how to present a one-page board view. It assumes you already run a programme; for the operating model see the Human Risk Management Programme Guide, and for survey design see the Security Culture Measurement Guide. It is written for awareness leads, human risk analysts and CISOs who have to defend their numbers in front of an audit committee.
Key takeaways
- Tier every metric. Activity, knowledge, behaviour and outcome metrics answer different questions; only behaviour and outcome metrics belong in front of the board.
- Specify before you measure. A metric without a written numerator, denominator, data source, exclusions and owner will be calculated differently each quarter and lose credibility at the first challenge.
- Separate KPIs from KRIs. KPIs show the programme is working; KRIs show that human-enabled risk is moving towards or beyond appetite, and each KRI needs an amber and red threshold agreed in advance.
- Prefer control-backed measures. A behaviour that is also enforced or logged by a control (payment call-back, help desk identity check, phishing-resistant MFA) produces evidence that does not depend on simulations.
- Protect the people in the data. Report at team level, suppress small groups and agree data use with HR and privacy before the first dashboard, or the reporting behaviour you are measuring will fall.
Four tiers of awareness metric
NIST SP 800-50 Rev. 1 lists quantitative measures that range from attendance and completion to employee incident reporting and longitudinal behaviour change, such as timely operating system updates and MFA adoption, plus qualitative measures such as attitude surveys, focus groups and interviews. The useful step is to sort those measures by what they can prove.
| Tier | Question it answers | Examples | Audience |
|---|---|---|---|
| 1. Activity | Did the programme run? | Completion by role path, campaigns sent, cost per participant, content refresh date | Programme team, auditors |
| 2. Knowledge and attitude | Do people know and believe the right things? | Pre and post knowledge tests (NIST suggests repeating at three months), survey index scores | Programme team, HR |
| 3. Behaviour | Do people act securely when it matters? | Report rate, time to first report, payment call-back compliance, MFA prompt denials reported, DLP overrides with valid justification | CISO, risk committee |
| 4. Outcome | Is human-enabled loss less likely or smaller? | Share of confirmed incidents first detected by a user, human-initiated incidents per 1,000 staff, dwell time on user-reported incidents | Board, audit committee |
Tier 1 is compliance evidence and stays in the appendix. Tier 2 explains movement in tiers 3 and 4 but should never be reported as success on its own. If a programme can only produce tier 1 and 2 numbers, that is itself a finding: the Security Awareness Maturity Model places completion-only evidence at Level 2.
Writing a metric that survives challenge
Every reported metric needs a specification. Keep it in your metrics dictionary (the Security Metrics Dictionary Template has the full field list) and change it only through versioned updates.
- Name and tier: for example "Real phish first reported by a user (outcome)".
- Numerator and denominator: exact counting rules. "Confirmed malicious email incidents where the earliest signal was a user report" divided by "all confirmed malicious email incidents in the period".
- Population and exclusions: which staff, mailboxes and channels; exclude shared and service mailboxes, staff on leave, and simulation messages from real-incident metrics.
- Source system and field: the report button telemetry, the SOC case tool and the field that marks a case as user-originated.
- Frequency and lag: monthly, closed ten working days after month end so late-confirmed incidents are counted.
- Direction and thresholds: higher is better; amber and red levels if it is a KRI.
- Owner and reviewer: one person produces it, another checks it each quarter.
- Known distortions: what can move the number without behaviour changing (a new mail filter, a reorganisation, a change of report button).
Board KPIs and KRIs
A board needs fewer numbers than a programme team. The set below is designed for quarterly reporting; the targets and thresholds in square brackets are placeholders to set from your own baseline, not benchmarks.
| Measure | Type | Definition | Primary source | Threshold (example) |
|---|---|---|---|---|
| User-first detection share | KPI | Confirmed phishing and social engineering incidents where a user report was the first signal, as a share of all such incidents | SOC case management | Target [rising quarter on quarter] |
| Median time to first report (real campaigns) | KPI | Minutes from first delivery of a confirmed malicious campaign to the first user report, median across campaigns in the quarter | Message trace plus report telemetry | Target [below baseline] |
| Human-initiated incidents per 1,000 staff | KRI | Confirmed incidents with a human action as the enabling step (credential entry, payment release, MFA approval, data upload) per 1,000 staff | Incident records with a root-cause field | Amber [baseline plus 20%], red [baseline plus 50%] |
| High-risk process bypass rate | KRI | Payment detail changes and help desk credential resets completed without the required verification step, as a share of all such events | Finance workflow and service desk audit trail | Amber [any in a quarter], red [any linked to an incident] |
| Repeat high-risk behaviour | KRI | Share of staff with two or more high-risk events of the same type in 90 days, by business unit | Human risk data store | Amber and red set per unit size |
| Control-backed coverage | KPI | Share of catalogue behaviours that are enforced or logged by a control, not only taught | Behaviour catalogue | Target [rising to agreed level] |
The high-risk process bypass rate earns its place because help desk and payment verification failures have direct loss paths. In the MGM Resorts breach, attackers reportedly called the IT help desk posing as an employee to obtain access; a bypass metric drawn from the reset audit trail is the earliest warning that a verification step is being skipped. For how KRIs connect to appetite statements, see the Key Risk Indicators for Cyber Risk Guide.
Scenario-led measures beyond email
Email simulation metrics cover one channel. Build at least one behaviour metric and one outcome metric for each human-enabled scenario in your risk register.
| Scenario | Behaviour metric (leading) | Outcome metric (lagging) |
|---|---|---|
| Business email compromise and supplier fraud | Call-back verification completed on bank detail changes | Fraudulent payments attempted and released, value recovered |
| Help desk impersonation | Resets completed with the full identity check | Resets later linked to account takeover |
| MFA fatigue and push bombing | Unexpected prompts denied and reported | Sign-ins approved after repeated denials |
| Data to unapproved AI tools | DLP prompts that end in a cancelled upload | Confirmed data exposure events to unsanctioned services |
| Secrets in code and chat | Developer self-reports of an exposed key | Median hours from exposure to revocation |
| Voice and video impersonation | Out-of-band verification logged for executive requests | Requests actioned without verification |
Behaviour metrics rise before outcome metrics fall, so expect a lag of one or two quarters and say so in the first board paper. Role-specific content for these scenarios is in the Business Email Compromise Awareness for Finance Teams Guide and the Vishing and Help Desk Social Engineering Awareness Guide.
Calculation rules that keep numbers honest
- Band by difficulty: NIST SP 800-50 Rev. 1 points to the NIST Phish Scale (Technical Note 2276) for rating how hard a simulated email is to detect. Report simulation results within difficulty bands and never average across them.
- Separate real and simulated: keep simulation reports and real-threat reports in different measures. A board metric built on real campaigns cannot be gamed by choosing easier lures.
- Use medians for time: one report six hours late distorts a mean; publish the median and the 90th percentile.
- Suppress small groups: do not show a team-level rate for fewer than [10] people. It identifies individuals and swings wildly.
- Fix the denominator: report rate uses recipients who received the message, not all staff; incidents per 1,000 staff uses average headcount for the period.
- Watch report accuracy too: track the share of user reports that are confirmed malicious. A falling share with rising volume is healthy noise; a falling volume is the warning sign.
- Annotate breaks: mark the chart when a filter, report button or campaign design changes, and do not compare across the break without saying so.
Simulation-specific clean-up, such as filtering scanner clicks, is covered in the Phishing Simulation Programme Guide.
Data sources and pipeline
| Source | Feeds | Quality check |
|---|---|---|
| Report button and reporting mailbox | Report rate, time to first report, report accuracy | Reconcile counts with the mailbox monthly; confirm the button is deployed to all clients |
| SOC case management | User-first detection share, human-initiated incidents | A mandatory "first signal" and "enabling human action" field on every closed case |
| Identity provider logs | MFA denials, approvals after denials, phishing-resistant enrolment | Join on immutable user ID, not display name |
| Finance and service desk workflows | Process bypass rate | Sample ten records a quarter against the audit trail |
| DLP and secure web gateway | Cancelled uploads, unsanctioned AI use | Exclude test accounts and known false-positive rules |
| Secret scanning | Exposure to revocation time, self-reports | Revocation timestamp from the issuing system, not the ticket |
| Learning platform and surveys | Tier 1 and 2 context | Survey response rate reported alongside every index |
Key everything to the identity in your identity provider and HR record, so moves between teams follow the person and leavers drop out of denominators.
Setting baselines, targets and thresholds
- Baseline: collect at least two quarters of data without changing interventions, or one quarter plus historical incident data, before setting any target.
- Agree direction first: for each measure, agree which way is good and what could move it falsely. Publish this before the first number.
- Set KRI thresholds from appetite: tie amber and red to the organisation's risk appetite statement, using the Cyber Risk Appetite Statement Template as the anchor, and name who acts at each level.
- Pilot the pack: run one quarter as a shadow report to the CISO and risk lead. Fix definitions that produced arguments.
- Go live: present to the risk or audit committee with the definitions annex.
- Roll back cleanly: if a measure proves unreliable, withdraw it, state why in the next pack and keep the history; never silently redefine it.
The one-page board view
NIST SP 800-50 Rev. 1 expects programme managers to give senior leadership regular reports on goals, objectives and performance metrics. A page that works has four blocks:
- Headline: one sentence on whether human-enabled risk is within appetite, with the KRI status.
- Trend panel: the six measures above as eight-quarter sparklines with thresholds drawn in.
- Exceptions: any amber or red KRI with cause, owner and date for the next reading.
- Decisions asked: funding or process changes needed, such as making call-back mandatory for a subsidiary.
Leave completion rates, quiz scores and team league tables out of the board page. For the wider pack this page sits in, see the Board Cyber Risk Reporting Guide.
Small and large organisations
- Under about 500 staff: track four measures in a spreadsheet: user-first detection share, median time to first report, process bypass rate and repeat high-risk behaviour. Report twice a year. Team-level views will usually fall below the suppression threshold, so report organisation-wide.
- Mid-sized: add business unit views, the scenario table for your top three scenarios and quarterly reporting.
- Large or regulated: automate the pipeline into the SIEM or data platform, add independent quarterly checking of each metric and a second line review of KRI thresholds.
Privacy and fairness
Behaviour metrics are workforce monitoring. The NCSC warns that metrics signal what an organisation values and that a blame culture stops people reporting; a programme that publishes names will see its report rate fall. In the UK, the ICO's guidance on monitoring workers applies, and the ICO states that guidance is under review following the Data (Use and Access) Act, so check the current version and complete a data protection impact assessment before individual-level scoring. Agree with HR that individual data drives coaching only, set a retention period and publish both in the programme policy.
Where identity fits
Several of the strongest measures are identity events: MFA approvals after denials, help desk resets and phishing-resistant enrolment. Those come from identity provider logs, so the identity team should co-own them. Developers create non-human identity risk through human behaviour when API keys and tokens leak, which makes exposure to revocation time a natural shared metric with platform engineering. See the Identity Security Metrics Guide for the identity-side measures.
Framework mapping
| Framework | Reference | Relevant metric evidence |
|---|---|---|
| NIST CSF 2.0 | PR.AT-01, PR.AT-02 | Coverage by role path, behaviour metrics for general and specialised roles |
| NIST CSF 2.0 | GV.RR-04 | HR-linked data, retention and use rules |
| NIST CSF 2.0 | GV.OV-03, ID.IM-01 | Board review of KPIs and KRIs and the improvements that follow |
| NIST SP 800-53 Rev. 5 | AT-2, AT-2(3), AT-3, AT-4 | Literacy and social engineering training, role-based training, training records |
| NIST SP 800-53 Rev. 5 | IR-6 | User incident reporting measures |
| CIS Controls v8.1 | Control 14 | Programme intended to influence workforce behaviour; see the CIS Control 14 Security Awareness and Skills Training Guide |
Common failure patterns
- Click rate as the headline: it rewards easy lures and says nothing about reporting. Show it, banded by difficulty, only in the appendix.
- Undefined metrics: two analysts produce different report rates from the same quarter, and the board stops trusting the page.
- KRIs with no thresholds: a trend line nobody is obliged to act on is decoration.
- Only email measured: voice, help desk and payment paths go unmeasured while they carry the largest losses.
- Silent redefinition: changing a denominator mid-year to improve a trend destroys the series.
- Named individuals in dashboards: managers see who clicked, word spreads and self-reporting stops.
Practitioner checklist
- Classify every current metric into activity, knowledge, behaviour or outcome.
- Write a specification for each reported measure in the metrics dictionary.
- Pick no more than six board measures and label each KPI or KRI.
- Set amber and red thresholds for every KRI from the risk appetite statement.
- Add "first signal" and "enabling human action" fields to SOC case closure.
- Build one behaviour and one outcome metric per human-enabled scenario.
- Band simulation results by Phish Scale difficulty and keep them out of board KPIs.
- Apply a minimum group size and agree data use with HR and privacy.
- Run one shadow quarter before the first committee report.
Standards and references
- NIST: SP 800-50 Rev. 1, Building a Cybersecurity and Privacy Learning Program
- NIST: TN 2276, NIST Phish Scale User Guide
- NIST: Cybersecurity Framework 2.0
- NIST: SP 800-53 Rev. 5
- NCSC: Phishing attacks, defending your organisation
- NCSC: Cyber security culture principles
- CIS: Control 14, Security Awareness and Skills Training
- ICO: Employment practices and data protection, monitoring workers
Related NHI Mgmt Group resources: Human Risk Management Programme Guide · Security Culture Measurement Guide · Security Awareness Maturity Model · Security Metrics and KPIs Programme Guide