Topic starter
01/08/2026 10:43 pm
TL;DR: Scanner accuracy in AppSec is often distorted by precision or recall claims, but Checkmarx Zero research across seven production codebases shows why the F1 score is a better benchmark, with its hybrid SAST scoring 0.64 versus about 0.20 for a standalone LLM reviewer. The governance lesson is that noisy detection can be as dangerous as missed vulnerabilities because developer trust collapses when findings are unbalanced.
Editorial analysis by NHI Mgmt Group, based on content published by Checkmarx: “Your Scanner’s Accuracy Claims Are Only Half the Story”.
By the numbers:
- The study compared 747 findings across the benchmark set.
- The standalone LLM reviewer had a 44.1% false positive rate.
- Checkmarx SAST scored an F1 score of 0.64 versus approximately 0.20 for Claude Opus 4.7.
Key questions
Q: What breaks when AppSec scanners optimise for precision or recall alone?
A: They create a false sense of accuracy.
Q: Why do AppSec scanner claims need both precision and recall?
A: Because each metric hides the failure mode of the other.
Q: How can security teams tell whether scanner noise is becoming a governance problem?
A: Look for a drop in investigation quality, slower triage, and developers starting to ignore findings.
Practitioner guidance
- Require F1 score in vendor evaluations Ask every AppSec scanner vendor for F1, precision, and recall on a named production dataset, not a synthetic benchmark.
- Test against real codebases Run pilots on code that resembles your own production environment, because accuracy on toy projects does not predict how the tool will behave in messy, dependency-heavy repositories.
- Measure developer trust in findings Track whether investigators are dismissing findings because of noise, since trust collapse is often the first operational sign that a scanner has crossed from useful control into ignored churn.
Bottom line: AppSec scanner performance cannot be judged fairly when precision and recall are reported in isolation, because each can improve while the overall control weakens.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
01/08/2026 10:52 pm
F1 score is the right governance lens for scanner accuracy because it exposes the tradeoff vendors most want to obscure. Precision-only claims can mask missed vulnerabilities, while recall-only claims can mask developer overload. In governance terms, that is not a reporting preference but a control failure, because the organisation cannot tell whether it is buying detection or noise. The practitioner conclusion is to treat balanced measurement as part of security assurance, not marketing.
A few things that frame the scale:
- Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap, according to the State of Secrets in AppSec.
A question worth separating out:
Q: Should teams prefer real production datasets over benchmark samples when evaluating scanners?
A: Yes. Production datasets reveal how a scanner behaves in the conditions that matter, including messy dependencies, complex code paths, and mixed-quality findings. Synthetic benchmarks are useful for comparison, but they can overstate accuracy and hide the cost of false positives or missed vulnerabilities in live environments.
👉 Read our full editorial: F1 score exposes the real tradeoff in AppSec scanner accuracy