What are the signs that an application security KPI programme is not working?

A weak KPI programme shows up when teams cannot explain why a metric matters, when the same issues recur release after release, or when reporting looks healthy while risk remains unchanged. Another warning sign is overreliance on vanity metrics, such as scan counts, that do not reveal remediation progress, control coverage, or whether secure engineering habits are improving.

Why a KPI programme fails to tell you anything useful

A KPI programme stops being useful when the metric is easy to report but hard to act on. That usually means the measure is disconnected from a control objective, so teams can keep presenting green dashboards while the underlying weakness persists. In appsec, the problem is rarely measurement volume; it is whether the metric tracks a decision, a behaviour change, or a security outcome.

Healthy-looking charts can also hide stagnation. If the same findings recur, the same exceptions are repeatedly accepted, or the metric never changes the conversation with engineering, the KPI is functioning as reporting theatre rather than management signal. That is especially true when scan counts are elevated to the status of progress, even though they say little about remediation quality, coverage, or risk reduction.

Metrics that focus only on activity often miss the point. A programme can show more scans, more tickets, or more reviews without improving the security posture that matters, such as faster fix rates, reduced recurrence, stronger control coverage, or better secure-by-default development habits.

How to recognise a vanity-metric programme

Vanity metrics are the clearest warning sign because they answer “how much did we do?” rather than “did security improve?”. In practice, that often means counts of findings, scans, or assessments are reported without context on severity, closure, aging, re-open rates, or whether the metric is tied to a decision threshold. A useful KPI should make it obvious what action follows when the number moves.

Another sign is metric drift. If a KPI started as a way to improve remediation but is now mainly used to defend a dashboard, people begin optimising the number instead of the system. That can produce behaviours like suppressing findings, reclassifying issues, or chasing easy-to-close tickets while ignoring deeper design flaws.

Look for weak signal quality too. If the measure cannot distinguish between a team that fixed root causes and a team that merely closed low-value items, it is not helping leaders allocate effort. For application security, the better question is whether the metric shows progress in reducing exposure and improving engineering practice, not just whether the pipeline produced more output.

Compare the KPI against a specific control objective, not against last quarter’s volume.
Check whether the metric changes prioritisation, resourcing, or release decisions.
Test whether the number can improve while residual risk stays the same.

Risk and Threat Considerations

When appsec KPIs are weak, the main risk is false confidence. Leaders may believe control performance is improving because reports are consistent and dashboards are positive, while exploitable weaknesses remain open, repeated, or poorly governed. Over time, that can normalise underinvestment in remediation and allow recurring defects to become an accepted part of delivery.

Failure mechanism: The programme measures output instead of control effectiveness, so recurring defects, slow remediation, and poor fix quality are not visible in the KPI set. Teams then optimise the report, not the risk reduction process.

Impact: Security work is misallocated, real exposure persists, and the organisation loses the ability to tell whether its appsec investment is reducing attack surface or only generating activity.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST CSF 2.0 and CIS Controls v8 set the governance and control requirements practitioners need to meet.

Framework	Control / Reference	Relevance
OWASP Non-Human Identity Top 10	NHI-01 — Secrets and Credential Management	Appsec KPIs often fail when they ignore secret exposure and rotation progress.
Recommendation — Track secret rotation and exposure remediation, not just scan counts.
NIST CSF 2.0	GV.OC-01 — Organisational Context	KPIs must reflect how the organisation defines security outcomes and decisions.
Recommendation — Tie each KPI to an explicit security objective and decision owner.
CIS Controls v8	6.1 — Establish and Maintain a Secure Application Development Process	A KPI programme is weak when it does not measure secure-development behaviour change.
Recommendation — Measure secure-development outcomes alongside activity counts.

Practitioner Guidance

What to verify: Each KPI should have a named consumer, a decision it informs, and an observable follow-up action when it moves. If nobody can explain why the number matters to engineering or security leadership, it is not yet a management KPI.

What to measure: Prefer measures that show remediation progress and control quality, such as time to remediate, recurrence rates, control coverage, and the share of findings fixed at the root cause. If the programme tracks only scan volume or issue count, it is likely measuring noise rather than improvement.

Common mistake: Treating dashboard consistency as programme health. A stable report can still describe a stagnant or deteriorating security posture, so the question is whether the KPI changes behaviour, not whether it is easy to present.

Practitioner takeaway: A good appsec KPI programme makes weak security hard to ignore and improvement hard to fake; if the metric can look good while the same problems keep returning, it is not governing risk.