Vanity Metric

Expanded Definition

A vanity metric is a measurement that creates the appearance of progress without proving that security outcomes improved. In NHI governance, that often means reporting counts such as training completions, policy acknowledgements, or dashboard activity while leaving access paths, secret exposure, and ownership gaps unchanged. The result is a metric that is easy to present, but weak as evidence.

Definitions vary across vendors and programmes, so the key test is whether the metric is tied to a control objective, a baseline, and a change in risk. A useful benchmark should answer whether access was reduced, secrets were rotated, privileges were removed, or orphaned identities were remediated. That is why NHI Management Group treats outcome linkage as the core discipline, not display value. The same issue appears in broader governance models such as the NIST Cybersecurity Framework 2.0, where measurements should support improved risk management rather than simple reporting volume.

The most common misapplication is treating participation counts as evidence of control effectiveness, which occurs when teams report activity without verifying any change in access, secrets, or exposure.

Examples and Use Cases

Implementing measurement rigorously often introduces reporting friction, requiring organisations to weigh easy-to-collect activity data against harder but more meaningful outcome evidence.

• A team reports 100% completion for an API key review campaign, but no keys were rotated and no stale credentials were removed.
• A dashboard shows a decline in open tickets after an NHI cleanup sprint, yet Ultimate Guide to NHIs notes that weak visibility and poor rotation remain common in the field, so the risk picture may not have changed.
• An identity programme celebrates the number of service accounts inventoried, while privileged entitlements and embedded secrets still sit outside a secrets manager.
• A compliance report highlights policy acknowledgements for NIST Cybersecurity Framework 2.0, but no evidence shows that least privilege or segregation of duties improved.
• An engineering team tracks how many agents were onboarded to a platform, but not whether their tool access was constrained or monitored.

In practice, vanity metrics are often useful only as leading indicators of engagement, not as proof of control performance. For NHI security, the better question is whether the metric changes when secrets are removed, privileges are tightened, or ownership is corrected.

Why It Matters in NHI Security

Vanity metrics are dangerous in NHI environments because they can mask the very conditions that create breach exposure. NHI Management Group reports that 80% of identity breaches involved compromised non-human identities such as service accounts and API keys, and 96% of organisations store secrets outside secrets managers in vulnerable locations including code, config files, and CI/CD tools, as documented in the Ultimate Guide to NHIs. Those numbers matter because they point to control failure, not dashboard success.

A vanity metric can make a programme look mature while excessive privileges, stale keys, and orphaned accounts remain untouched. It also weakens executive decision-making, because leaders may fund activities that are easy to count instead of controls that reduce attack surface. Proper governance requires pairing every reported number with a security consequence, such as revoked access, rotated credentials, or restored ownership. That discipline aligns with the risk-based measurement approach in the NIST Cybersecurity Framework 2.0.

Organisations typically encounter the cost of vanity metrics only after a secrets leak, privilege abuse, or audit failure reveals that the reported progress never translated into actual control improvement.

Related resources from NHI Mgmt Group

• How can organisations use a security assessment without turning it into a vanity metric?
• What is the difference between an outcome metric and a control metric?
• How do you know if an identity metric is actually working?
• Why do AI governance programmes need a single readiness metric?