Notifications
Clear all
Topic starter
01/08/2026 10:43 pm
TL;DR: Scanner accuracy in AppSec is often distorted by precision or recall claims, but Checkmarx Zero research across seven production codebases shows why the F1 score is a better benchmark, with its hybrid SAST scoring 0.64 versus about 0.20 for a standalone LLM reviewer. The governance lesson is that noisy detection can be as dangerous as missed vulnerabilities because developer trust collapses when findings are unbalanced.
NHIMG editorial — based on content published by Checkmarx: F1 score exposes the real tradeoff in AppSec scanner accuracy
By the numbers:
- Checkmarx SAST scored an F1 score of 0.64 across seven production codebases, compared with approximately 0.20 for Claude Opus 4.7 as a standalone LLM reviewer.
- The benchmark covered 747 findings across seven real production codebases in Python, Go, and C#.
- Claude Opus 4.7 generated a 44.1% false positive rate, meaning nearly half its findings were wrong.
Questions worth separating out
Q: How should security teams evaluate AppSec scanners without being misled by vendor metrics?
A: Use F1 score as the baseline, then verify the underlying precision, recall, and dataset.
Q: Why do precision and recall create blind spots in scanner selection?
A: Precision can look strong when a scanner simply reports fewer findings, while recall can look strong when it flags almost everything.
Q: What do security teams get wrong about low false positive rates?
A: They often assume a lower false positive rate means better security, but it may just mean the scanner is missing more issues.
Practitioner guidance
- Require F1 score disclosure Ask every scanner vendor for an F1 score measured on real production code, not only on synthetic benchmarks or curated demo projects.
- Separate precision and recall in evaluation Insist on precision and recall being reported together so one metric cannot conceal weakness in the other.
- Test against your own codebase Run the scanner on representative production repositories that include real secrets, auth patterns, and application logic that mirror your environment.
What's in the full article
Checkmarx's full analysis covers the operational detail this post intentionally leaves for the source:
- Benchmark methodology across seven production codebases and how the test set was assembled
- The breakdown of precision, recall, and F1 for the hybrid SAST approach versus the LLM reviewer
- How the classification layer changes which findings reach analysts in practice
- The vendor's own guidance on using F1 when comparing scanner candidates
👉 Read Checkmarx's analysis of F1 score and AppSec scanner accuracy →
AppSec scanner F1 score: what precision and recall hide?
Explore further
01/08/2026 10:52 pm
F1 score is a governance control, not just a benchmark: scanner accuracy metrics shape whether developers trust AppSec output enough to act on it. A metric that rewards silence or noise rather than balanced detection creates governance debt in the backlog. For programmes that touch secrets, service credentials, and other NHI-adjacent application risks, the control question is whether the tool produces decisions that developers can sustain.
A question worth separating out:
Q: Who is accountable when scanner metrics hide real vulnerabilities?
A: Accountability sits with the security team and the procurement process, not just the vendor. If a programme accepts unnamed datasets, single-metric claims, or untested synthetic benchmarks, it inherits the risk of blind spots. Governance needs clear measurement criteria before a scanner is adopted or expanded.
👉 Read our full editorial: F1 score exposes the real tradeoff in AppSec scanner accuracy
