Why do application security programmes struggle to prove improvement even when teams are shipping fixes?

Why shipment does not equal measurable improvement in application security

application security programmes often struggle to prove progress because output and outcome are easy to confuse. Teams can close findings, but if new issues are discovered at the same pace, the backlog may look flatter without any real reduction in exposure. Measurement becomes especially weak when reporting is built around raw counts rather than aging, severity mix, repeat findings, and whether the same weaknesses keep reappearing in the same applications. For a practical control perspective, ISO/IEC 27002:2022 Information Security Controls is useful because it reinforces that security improvement has to be evidenced through control operation, not assumed from activity volume alone.

What teams usually need is a view that separates newly found risk from risk that is actually being retired. Without that separation, leaders may reward teams for clearing easy items while the highest-risk issues continue to age. In practice, many security teams encounter this only after reporting has already normalised “closed tickets” as a proxy for reduced exposure.

How remediation metrics need to be structured to show real movement

Good application security measurement connects discovery, remediation, and residual exposure into one timeline. A fix is only meaningful if the programme can show how quickly issues move from open to closed, how many remain open beyond an acceptable threshold, and whether the same control failure keeps surfacing. That means tracking more than count-based metrics. The programme should distinguish aged vulnerabilities, reopened findings, accepted exceptions, and items that were truly eliminated through code or configuration change.

Teams also need to report by context, because a single aggregate number can hide where risk concentrates. A small number of high-severity issues in a critical service may matter more than a large number of low-risk issues in a low-impact application. Likewise, a programme can appear to improve if it closes many simple findings while failing to reduce risk in the applications that matter most. That is why trend lines, severity bands, and application ownership all belong in the same view.

A useful operational model is to pair remediation throughput with exposure age. Throughput shows how much work teams complete; exposure age shows how long the organisation remains vulnerable. When those two curves move together, progress is more credible. When throughput rises but age does not fall, the programme may be generating motion rather than risk reduction.

• Track open findings, closed findings, and reopened findings separately.
• Break results down by severity, application, and exception status.
• Use aging and time-to-remediate to reveal whether fixes are keeping pace with discovery.
• Watch for repeated defect patterns, because recurring classes of issues often indicate control failure rather than isolated mistakes.

This guidance breaks down when the organisation cannot link findings to stable application ownership or cannot distinguish duplicate reports from genuine re-exposure.

Where measurement gets distorted, and what the edge cases usually mean

Tighter reporting often increases administrative overhead, requiring organisations to balance measurement precision against the effort needed to keep data clean. That tradeoff matters because some programmes overfit to scorekeeping and lose sight of the underlying risk picture. A falling issue count may simply reflect a backlog purge, a scanning scope reduction, or a temporary freeze on new findings, none of which proves that security has improved.

There is also a genuine consensus gap in the industry about how to compare teams fairly. Some groups favour velocity-style metrics, while others argue that only exposure-based metrics should be used for leadership decisions. The practical answer is that neither alone is sufficient. Velocity helps explain operational capacity, but exposure age and severity mix are what show whether the organisation is becoming safer. If a metric cannot survive a change in application count, scanning depth, or reporting window, it should not be used as a headline measure of improvement.

For complex programmes, the most important edge case is risk acceptance. A fix may be shipped, but if the issue is converted into an exception, deferred behind a compensating control, or reopened in a different component, the headline closure no longer means the risk is gone. The programme should treat these cases as lifecycle states, not as clean remediation wins.

Practitioner Guidance

What to prioritise: Focus first on whether your reporting can show exposure age, not just closure volume. If the organisation cannot see how long material issues stay open, it cannot tell whether fixes are reducing risk or merely changing the shape of the backlog.

What to verify: Check that closed items really represent eliminated risk, not duplicates, scope changes, compensating controls, or deferred exceptions. The strongest indicator of real improvement is a decline in aged high-severity exposure across critical applications, not a one-month drop in total tickets.

What practitioners underestimate: Repeated findings are often more informative than raw counts because they reveal whether secure coding, review, testing, or release controls are actually changing behaviour. A programme that keeps fixing the same defect class is improving throughput, but not necessarily improving security.

Practitioner takeaway: Leaders should judge application security progress by how quickly meaningful risk leaves the system and whether the same weaknesses stop recurring, because closure volume alone can hide a stagnant exposure profile.

Related resources from NHI Mgmt Group

• Why do application security programs struggle to prove value to leadership even when testing is happening?
• Why do small security teams struggle with cloud detections even when they have modern tools?
• How do security teams stop the same application vulnerability from shipping twice?
• Why do CTEM programmes fail even when teams buy more security tools?