Because scans, tickets, and alerts measure work volume, not whether risk actually falls. A team can generate more output while leaving blind spots, noisy detections, and overdue fixes untouched. Outcome KPIs reveal whether the programme changes engineering behaviour and reduces exposure.
Why activity looks like progress even when risk stays flat
Activity metrics are easy to count, easy to trend, and easy to present, which is exactly why they can mislead AppSec leaders. A rising scan count, ticket count, or alert count often reflects more throughput rather than lower exposure. That is why teams can look busier while the underlying control gaps remain unchanged.
The problem is not that scans, tickets, and alerts are useless. The problem is that they are process indicators, not security outcomes. They tell you whether work is happening, but not whether the work changes developer behaviour, closes exploitable paths, or reduces the likelihood of a real incident.
For application security, that distinction matters because risk reduction depends on what gets fixed, prevented, or removed, not on how much attention the programme attracts. A mature programme should be able to show fewer repeated findings, shorter exposure windows, and less reintroduction of the same weakness across releases.
What activity metrics hide in practice
Activity-heavy reporting can conceal several common failures. First, it can overstate coverage when scanners run often but miss the most important paths, so blind spots survive beneath the volume. Second, it can reward teams for generating findings faster than engineering can absorb them, which creates backlog without changing risk. Third, it can make noise look like vigilance, even when alert quality is poor or triage is shallow.
This is why output metrics need to be interpreted alongside evidence of control effectiveness. In a secure software programme, OWASP SAMM is useful because it frames maturity around practices and capability growth, not just operational volume. Likewise, OWASP ASVS helps anchor measurement to what a system must actually satisfy, rather than how many checks were performed.
Practitioners should also be careful with “closed” ticket counts. A ticket can be closed because a fix shipped, because the issue was accepted, or because the finding was reclassified. Those are very different outcomes. Without lifecycle context, ticket closure is a weak proxy for genuine exposure reduction.
What to measure instead of raw volume
The better question is whether the programme is changing the security state of applications and the behaviour of the teams that build them. Outcome measures usually answer that better than activity measures. Examples include the percentage of high-severity issues fixed within target time, the recurrence rate of the same weakness, the share of findings caught before production, and the proportion of critical components that meet baseline requirements.
A useful appsec maturity view also tracks whether controls are becoming embedded in the delivery process. That means measuring whether issues are prevented earlier, whether risky changes are caught before release, and whether findings decrease in repeatability over time. OWASP Top 10 remains a practical reference for anchoring those outcome discussions to the risk classes developers actually need to avoid, while NIST SSDF (SP 800-218) supports the shift from ad hoc checking to repeatable secure development practice.
Good measurement also distinguishes between detection and remediation. Finding more issues is not maturity if the fix rate, age distribution, and re-open rate stay poor. The useful signal is whether the team is shrinking exposure faster than the environment is producing new defects.
Risk and Threat Considerations
Activity metrics can create a false control narrative, which is dangerous because it delays hard decisions. Teams may believe they have improved simply because review queues are full, but attackers only benefit if exploitable weaknesses remain reachable. High output with stagnant fix quality can preserve old attack paths while giving leadership a comforting dashboard.
Failure mechanism: The programme measures throughput instead of exposure reduction, so repeated findings, slow remediation, and unowned weaknesses continue to accumulate behind a busy reporting layer.
Impact: Engineering teams may keep shipping into the same unresolved risk patterns, leadership may underinvest in the controls that actually reduce exposure, and attackers may still find the same exploitable conditions after the metrics have already turned green.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP SAMM, OWASP ASVS and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP SAMM | Software Assurance Maturity Model | Maturity is the core issue, not scan volume. |
| Recommendation — Use SAMM to assess whether security practices are maturing beyond activity output. | ||
| OWASP ASVS | V15 — Secure Coding and Architecture | Outcome metrics should reflect whether engineering controls reduce exposure. |
| Recommendation — Use ASVS to anchor metrics to verifiable security requirements and reduce gap noise. | ||
| NIST SP 800-53 Rev 5 | CA-7 — Continuous Monitoring | Activity metrics need monitoring that proves control effectiveness over time. |
| RA-5 — Vulnerability Monitoring and Scanning | Scans are only meaningful when they drive timely remediation and lower exposure. | |
| SI-2 — Flaw Remediation | The key maturity test is whether flaws are actually corrected and reintroduced less often. | |
| Recommendation — Measure whether monitoring produces actionable risk reduction, not just more findings. Track scan findings through remediation and verify they close risk, not just tickets. Use flaw-remediation data to confirm defects are fixed and recurrence is falling. | ||
Practitioner Guidance
What to prioritise: Replace the headline dashboard with a small set of outcome measures that tie directly to exposure, fix quality, and recurrence. If a metric does not help answer “did the application become safer?”, it should be treated as supporting context, not as a maturity claim.
What to verify: Check whether improvements hold across releases, not just within a reporting cycle. A genuine maturity signal is sustained reduction in repeat findings, shorter time to remediate critical issues, and fewer exceptions in the most exposed services.
Practitioner takeaway: Activity is a useful input, but maturity only exists when the control changes engineering behaviour and measurably reduces risk.