F1 score exposes the real tradeoff in AppSec scanner accuracy

At a glance

What this is: This article argues that AppSec scanner accuracy should be judged with F1 score, not precision or recall alone, because single-metric reporting can hide either missed vulnerabilities or excessive noise.

Why it matters: For AppSec, IAM, and platform security teams, the metric choice affects whether developers trust findings, whether real issues stay hidden, and how tools are evaluated before they reshape the remediation workflow.

By the numbers:

• The study compared 747 findings across the benchmark set.
• The standalone LLM reviewer had a 44.1% false positive rate.
• Checkmarx SAST scored an F1 score of 0.64 versus approximately 0.20 for Claude Opus 4.7.

Context

AppSec scanners are meant to reduce risk by surfacing real weaknesses, but the control breaks down when teams cannot tell whether a tool is missing issues or burying them in noise. Precision and recall each describe part of the picture, yet neither alone captures the operational tradeoff that determines whether developers will trust and act on findings.

That matters beyond application security because governance depends on usable signal. If findings are too noisy, remediation stalls and people stop looking; if findings are too sparse, real vulnerabilities remain hidden. The article also has a useful identity-adjacent lesson for NHI programmes: the same measurement problem appears whenever teams try to evaluate security tools against large, messy estates of credentials, secrets, or code paths.

What the article describes is typical of production security operations, not an edge case. Real environments are noisy, and any metric that hides that noise creates false confidence in the control.

Key questions

Q: What breaks when AppSec scanners optimise for precision or recall alone?

A: They create a false sense of accuracy. Precision-only tuning can suppress noise by missing real vulnerabilities, while recall-only tuning can find more issues by overwhelming developers with low-value findings. In both cases, the team loses a usable control because either real weaknesses are missed or the queue becomes too noisy to trust.

Q: Why do AppSec scanner claims need both precision and recall?

A: Because each metric hides the failure mode of the other. Precision tells you how many findings are real, while recall tells you how many real issues were found. If a vendor shows only one number, you cannot tell whether the scanner is actually balanced or simply trading missed vulnerabilities for fewer false alarms.

Q: How can security teams tell whether scanner noise is becoming a governance problem?

A: Look for a drop in investigation quality, slower triage, and developers starting to ignore findings. Those are the practical signs that the control has shifted from detection to background noise. At that point, the issue is not just tooling performance; it is loss of trust in the security workflow.

Q: Should teams prefer real production datasets over benchmark samples when evaluating scanners?

A: Yes. Production datasets reveal how a scanner behaves in the conditions that matter, including messy dependencies, complex code paths, and mixed-quality findings. Synthetic benchmarks are useful for comparison, but they can overstate accuracy and hide the cost of false positives or missed vulnerabilities in live environments.

Technical breakdown

Why precision and recall pull in opposite directions

Precision measures how many findings are actually real, while recall measures how many real issues a scanner finds. A tool can look strong on precision by reporting very little, or strong on recall by reporting almost everything, but neither state tells you whether the scanner is producing actionable security outcomes. In AppSec, that tradeoff matters because developer attention is a limited control surface. If a scanner optimises for one side only, the apparent improvement on the dashboard can conceal weaker protection in practice.

Practical implication: Evaluate scanner output as a balance of missed findings and false alarms, not as a single headline percentage.

Why benchmark data must match production reality

A score is only as useful as the dataset behind it. Purpose-built vulnerable samples often produce stronger results than real production code, where control flow, dependencies, and developer patterns are messier. That means a vendor quote without dataset context can be misleading even when the number itself is technically correct. The article’s central point is that accuracy claims need provenance, not just a metric. Without a named, representative codebase, teams cannot judge whether a scanner is performing against the environment they actually run.

Practical implication: Ask what codebase was tested before you accept any scanner accuracy claim.

NHI Mgmt Group analysis

F1 score is the right governance lens for scanner accuracy because it exposes the tradeoff vendors most want to obscure. Precision-only claims can mask missed vulnerabilities, while recall-only claims can mask developer overload. In governance terms, that is not a reporting preference but a control failure, because the organisation cannot tell whether it is buying detection or noise. The practitioner conclusion is to treat balanced measurement as part of security assurance, not marketing.

AppSec programmes fail when finding volume outpaces analyst and developer capacity. A scanner that floods queues trains teams to ignore alerts, which is a behavioural outcome that metrics alone can hide if they are incomplete. That is especially relevant to NHI and secrets workflows, where over-alerting can bury the exposures that actually matter. The practitioner conclusion is to align scanner evaluation with human consumption capacity, not just technical detection.

Production evidence matters more than benchmark theatre. Controlled datasets can demonstrate capability, but they do not substitute for real-world accuracy on shipped code. The broader lesson extends to AI-assisted security tooling: when the test environment is too clean, the resulting score can be directionally useful but operationally misleading. The practitioner conclusion is to demand named datasets and production context before changing tooling decisions.

False positives are not a cosmetic issue; they are a trust issue. Once teams learn that findings are usually wrong, they stop treating the scanner as a control and start treating it as background noise. That is the governance debt hidden behind many accuracy dashboards. The practitioner conclusion is to measure trust erosion as part of tool selection, not after deployment.

From our research library:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap, according to the State of Secrets in AppSec.
• Read next: NHI Security Platform Buyer's Guide

What this signals

Balanced measurement is the control, not the metric. Security teams should treat F1 score as a governance check on whether an AppSec scanner is actually usable in production. A tool that cannot show both precision and recall on real code is asking practitioners to trust a partial story, which is exactly how noise and missed findings become normalised.

Developer trust is the hidden dependency in AppSec tooling. Once teams learn that most findings are wrong, the scanner stops functioning as a control and becomes a queue-management problem. That same dynamic appears in NHI-adjacent tooling whenever noisy detection masks the real exposure set, because the organisation ends up responding to volume rather than risk.

For practitioners

• Require F1 score in vendor evaluations Ask every AppSec scanner vendor for F1, precision, and recall on a named production dataset, not a synthetic benchmark. If the dataset is undisclosed, treat the claim as incomplete rather than comparable.
• Test against real codebases Run pilots on code that resembles your own production environment, because accuracy on toy projects does not predict how the tool will behave in messy, dependency-heavy repositories.
• Measure developer trust in findings Track whether investigators are dismissing findings because of noise, since trust collapse is often the first operational sign that a scanner has crossed from useful control into ignored churn.
• Separate detection quality from volume Report false positives, false negatives, and action rate together so that a drop in one metric cannot be mistaken for overall improvement.

Key takeaways

• AppSec scanner performance cannot be judged fairly when precision and recall are reported in isolation, because each can improve while the overall control weakens.
• The article’s benchmark evidence shows why production code matters more than synthetic samples when deciding whether a scanner is fit for use.
• Teams should insist on balanced accuracy reporting before trusting any scanner to shape developer behaviour or remediation priorities.

Key terms

• Precision: The share of reported findings that are actually real weaknesses. High precision means fewer false positives, but it does not guarantee that the scanner is finding all of the important problems, so it must be read alongside recall to understand practical effectiveness.
• Recall: The share of real weaknesses that the scanner successfully identifies. High recall reduces the chance of missed vulnerabilities, but if it is achieved by over-reporting, teams can become overwhelmed and stop trusting the results.
• F1 Score: A combined metric that balances precision and recall in one number. In security scanning, it shows whether a tool is both finding real issues and avoiding excessive noise, which makes it more useful than either metric alone when comparing performance on real code.
• False Positive: A false positive is a scanner result that looks like a secret but is not actually sensitive. In secret governance, false positives matter because they consume analyst time, weaken trust in alerts, and can delay response to the findings that truly change exposure and access risk.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to the broader security workflows that depend on accurate detection and trustworthy lifecycle management.