What are the signs that a product security programme is measuring the wrong things?

Why Misaligned Security Metrics Distort Product Decisions

product security programmes go off track when the metrics they report are easy to count but weak at explaining whether engineering risk is actually falling. Vulnerability totals, closure speed, and similar lagging indicators can still be useful, but they often reward activity rather than reduction in exposure. The more serious problem is that teams may optimise for the dashboard instead of the control outcome.

That is why security leaders should prefer measures that connect to behaviour, coverage, and effectiveness, such as whether high-risk components are actually assessed, whether secure patterns are adopted, and whether repeat issues are shrinking over time. The EU Cyber Resilience Act is a useful reminder that product security is increasingly judged by evidence of control, not just evidence of reporting, and that matters even when the programme feels operationally busy. In practice, many security teams discover metric drift only after product roadmaps have already been shaped around the wrong signal.

What Healthy Product Security Measurement Looks Like

Good measurement starts with the security decision the programme needs to support. If the decision is prioritisation, the metric should show which teams, products, or control areas carry the most material exposure. If the decision is assurance, the metric should show whether required practices are actually happening. If the decision is improvement, the metric should show trend direction, not just current backlog.

Healthy programmes usually mix several layers of evidence. Leading indicators help show whether secure design and engineering practices are being adopted before defects appear. Coverage measures show whether the right assets, code paths, or product lines are actually inside scope. Quality measures show whether findings are meaningful, repeat issues are falling, and exceptions are being controlled rather than normalised. Outcome measures still matter, but they should be interpreted as part of a broader control picture, not as the whole story.

• Use metrics that map to a decision, not metrics that simply fill a report.
• Separate volume metrics from effectiveness metrics so one does not masquerade as the other.
• Track trends by product or team where that reveals control weakness, not just organisational totals.
• Check whether the metric can be gamed by superficial activity, because that is often the first sign it is the wrong measure.

For teams that need a control-oriented reference point, NIST SP 800-53 Rev 5 Security and Privacy Controls is useful when you want to anchor measurement to control intent rather than raw event counts.

Where this guidance breaks down is when a programme has no agreed risk model, because then even good metrics can be interpreted inconsistently.

Where Metrics Go Wrong in Real Programmes

Tighter measurement often increases reporting overhead, so organisations have to balance visibility against the cost of collecting, normalising, and interpreting the data.

One common failure is treating a low vulnerability count as proof of good security when the real explanation is incomplete scanning, narrow coverage, or weak triage. Another is relying on average remediation time when a small number of severe issues remain open for too long. A third is using organisation-wide rollups that hide large differences between product teams, release trains, or technology stacks. Consensus is not complete on the perfect product-security scorecard, but there is strong agreement that a useful metric must be actionable, stable enough to trend, and hard to game.

The right test is whether the metric would still be meaningful if a team improved the number without improving the underlying control. If the answer is no, the metric is probably measuring performance theatre rather than product security.

When product security is also tied to software component integrity or regulated release evidence, the EU Cyber Resilience Act becomes more relevant because it pushes organisations toward demonstrable security outcomes rather than broad assurances. The same logic is echoed in ISO/IEC 27002:2022 Information Security Controls, which helps teams distinguish control intent from superficial compliance reporting.

Risk and Threat Considerations

Wrong metrics create a governance risk because they can hide real exposure while making the programme look healthier than it is. They also create operational risk when teams optimise for measurable activity instead of reducing attack surface, repeat defects, or control gaps.

Failure mechanism: Lagging indicators and vanity metrics can be improved without changing the underlying security state, especially when coverage is partial, severity is distorted by triage rules, or teams learn to work around the measure.

Impact: Leaders may approve release decisions, staffing choices, or risk acceptance based on misleading evidence, leaving material weaknesses unaddressed and weakening accountability across engineering teams.

Practitioner Guidance

What to prioritise: Start by asking what decision each metric is meant to support. If a metric cannot drive prioritisation, assurance, or improvement, it should not be treated as a primary programme signal.

What to verify: Check whether the metric can be improved without any real security gain. If yes, pair it with a second measure that captures coverage, control adoption, or repeat-failure reduction so the dashboard cannot be gamed by activity alone.

Practitioner takeaway: The most useful product security metrics are the ones that expose control quality and decision quality, not the ones that merely prove the team has been busy.

Related resources from NHI Mgmt Group

• What are the signs that a product security programme is not ready for CRA compliance?
• What do security teams get wrong about measuring Zero Trust programmes?
• What do teams get wrong when they treat vulnerability scanning as a complete security programme?
• What do security teams get wrong about measuring application risk maturity?