How can organisations tell whether developers are actually using AppSec tools?

Why This Matters for Security Teams

Tool deployment does not prove tool usage. Security leaders often report full coverage because a scanner, SAST platform, or dependency check is enabled in the pipeline, yet developers may be bypassing findings, suppressing alerts, or routing changes around the control. The result is a false sense of assurance, especially when reporting focuses on licence counts, integrations, or raw alert volume rather than behaviour change.

This matters because AppSec only reduces risk when it affects design choices, pull request decisions, and remediation timing. A useful benchmark is whether the tool changes developer action before code is merged, not whether it produces a dashboard. NIST SP 800-53 Rev 5 Security and Privacy Controls provides a useful control lens for embedding security into development and change management, but current guidance still leaves room for organisations to define the operational metrics that prove adoption. In practice, many security teams discover non-usage only after a vulnerability becomes a production incident, rather than through intentional measurement of developer behaviour.

How It Works in Practice

To tell whether developers are actually using AppSec tools, teams need measures that connect tool output to workflow decisions. The key is to distinguish exposure from action. A scanner can be enabled on every repository, but if findings are routinely waived, ignored, or discovered only after merge, the control is present but not embedded.

Effective measurement usually combines adoption, effectiveness, and remediation signals:

• Adoption rate: how many active developers, repositories, or service teams trigger the tool in normal work.

• Pre-merge catch rate: how often issues are found before code reaches main branches or release candidates.

• Finding disposition: whether issues are fixed, accepted, suppressed, or deferred, and who approved that decision.

• Mean time to remediate by severity: whether high-risk issues are getting closed quickly enough to reduce exposure.

• Reopen rate: whether fixes are durable or only patching the immediate alert.

Those metrics work best when paired with identity-aware telemetry. If the tool can attribute actions to a developer, squad, or service account, organisations can see whether a specific group is consistently using the control or only complying during audits. For mature environments, the most valuable evidence is not a single dashboard but a trail showing tool invocation, review comments, override reasons, and remediation follow-through.

There is also a governance angle. Best practice is evolving, but many organisations now treat AppSec usage as a workflow control rather than a security reporting control. That means security teams should validate whether the tool is integrated into source control, CI/CD, and code review gates, and whether policy allows risky exceptions without strong justification. OWASP guidance and NIST SP 800-53 Rev 5 Security and Privacy Controls both support the idea that security controls must be measurable in operation, not just present in architecture.

These controls tend to break down when development is highly fragmented across many repositories and teams because local exceptions, custom pipelines, and inconsistent ticketing make the real usage path hard to observe.

Common Variations and Edge Cases

Tighter AppSec measurement often increases reporting overhead, requiring organisations to balance clear evidence of use against developer friction and engineering throughput.

Not every environment should be measured the same way. In some teams, a high pre-merge catch rate is healthy because the tool is surfacing issues early. In others, a high catch rate may simply mean code quality is poor and the control is compensating for upstream gaps. That is why guidance-vs-consensus matters here: there is no universal standard for the “right” percentage of findings fixed before merge.

Edge cases also matter. A developer may genuinely use the tool but repeatedly suppress findings because the rules are noisy or misaligned to the codebase. In that case, the problem is not adoption but control quality. Similarly, a mature team may appear to use AppSec tools less often because it has reduced change volume, moved to safer libraries, or shifted risk left into architectural reviews. The right interpretation depends on whether the tool is meant to block, guide, or inform decisions.

For cloud-native and agentic workflows, usage can also be indirect. A code assistant, CI bot, or repository policy engine may trigger AppSec checks on behalf of a developer, which means the organisation should measure the human decision chain, not just the human click. This is where identity and non-human identity governance intersect with AppSec measurement: if service accounts or automation are bypassing approval paths, usage data can look clean while the actual control is weak. CISA Secure by Design is useful here because it reinforces the principle that controls should be effective by default, not reliant on voluntary discipline alone.

Related resources from NHI Mgmt Group

• How can organisations tell whether NHI access for AI tools is actually controlled?
• How can organisations tell whether an appsec finding is actually high risk?
• How can organisations tell whether authentication is actually phishing-resistant?
• How can organisations tell whether AI tools are exposing data beyond policy intent?