F1 score exposes the real tradeoff in AppSec scanner accuracy

At a glance

What this is: This is an analysis of why AppSec scanner evaluation should use F1 score, not precision or recall in isolation, because the wrong metric can hide either missed vulnerabilities or overwhelming false positives.

Why it matters: IAM and security teams responsible for code risk, secrets exposure, and NHI-adjacent application controls need a metric that reflects operational trust, not just vendor-friendly summaries.

By the numbers:

• Checkmarx SAST scored an F1 score of 0.64 across seven production codebases, compared with approximately 0.20 for Claude Opus 4.7 as a standalone LLM reviewer.
• The benchmark covered 747 findings across seven real production codebases in Python, Go, and C#.
• Claude Opus 4.7 generated a 44.1% false positive rate, meaning nearly half its findings were wrong.

👉 Read Checkmarx's analysis of F1 score and AppSec scanner accuracy

Context

AppSec scanner evaluation is supposed to measure whether a tool finds real weaknesses without overwhelming teams with noise. In practice, many programmes inherit metrics that reward one side of that tradeoff and conceal the other, which is why accuracy claims can look stronger on paper than they behave in developer workflows. This matters for application security, secrets exposure, and any control that depends on teams trusting the output enough to act on it.

The primary governance issue is not whether a scanner produces a number, but whether that number reflects production reality. Precision alone can hide missed findings, while recall alone can hide noise. For teams managing identity-related application risk, including secrets, service credentials, and code that touches NHI workflows, the evaluation question is whether the tool improves decision quality or simply reshapes the backlog.

Key questions

Q: How should security teams evaluate AppSec scanners without being misled by vendor metrics?

A: Use F1 score as the baseline, then verify the underlying precision, recall, and dataset. A scanner that looks good on one metric can still miss real vulnerabilities or bury teams in noise. The right evaluation compares tools on production code that reflects your actual development environment, not on synthetic examples that flatter performance.

Q: Why do precision and recall create blind spots in scanner selection?

A: Precision can look strong when a scanner simply reports fewer findings, while recall can look strong when it flags almost everything. Either approach can hide real risk. Measuring both together is the only way to see whether a tool is actually balancing signal quality with detection coverage.

Q: What do security teams get wrong about low false positive rates?

A: They often assume a lower false positive rate means better security, but it may just mean the scanner is missing more issues. A cleaner queue is not helpful if real vulnerabilities never reach the analyst. Teams should judge whether the tool improves decision quality, not whether it makes the dashboard look calmer.

Q: Who is accountable when scanner metrics hide real vulnerabilities?

A: Accountability sits with the security team and the procurement process, not just the vendor. If a programme accepts unnamed datasets, single-metric claims, or untested synthetic benchmarks, it inherits the risk of blind spots. Governance needs clear measurement criteria before a scanner is adopted or expanded.

Technical breakdown

Precision, recall, and why each can mislead AppSec teams

Precision measures how many flagged findings are real, while recall measures how many real issues were actually found. Used alone, each metric can be gamed. A scanner can achieve high precision by reporting very little, which lets vulnerabilities slip through. It can achieve high recall by flagging almost everything, which floods analysts and developers with noise. The result is a false sense of confidence that looks tidy in dashboards but fails in operational use. In application security, this tension is especially sharp because codebases contain many paths, patterns, and edge cases that rule-based systems and AI reviewers handle differently.

Practical implication: require both precision and recall for any scanner you evaluate, not just the metric that flatters the product.

Why the F1 score is the more honest benchmark

The F1 score is the harmonic mean of precision and recall, so it penalises imbalance instead of averaging it away. That matters because a scanner with excellent precision and poor recall, or the reverse, still creates material risk. F1 forces both detection quality and signal quality into the same measure, making tradeoffs visible. It is not a universal truth because dataset choice still matters, but it is harder to obscure weaknesses with selective reporting. In AppSec, that makes F1 useful for comparing scanners against real production code rather than synthetic benchmarks that are too neat to reflect everyday complexity.

Practical implication: ask for F1 on production code, and treat unnamed or synthetic datasets as incomplete evidence.

How hybrid SAST architecture changes the measurement problem

Hybrid SAST combines deterministic pattern detection with model-based reasoning and a downstream classification layer. The deterministic layer catches known patterns, the AI layer expands coverage, and the classification stage reduces what reaches analysts. That architecture can improve balance, but it can also hide where quality is actually coming from if vendors report only one metric. The important point is that tool architecture affects both false positives and false negatives, so performance must be measured as a system, not as a single model. For teams reviewing scanner output, the question is whether the full pipeline improves trust in findings or merely repackages the same tradeoff.

Practical implication: test the full scanner pipeline against your own code so you can see whether filtering reduces noise without masking missed issues.

Threat narrative

Attacker objective: The objective is to exploit blind trust in misleading scanner metrics so real application weaknesses remain in production long enough to be abused.

1. Entry occurs when a scanner reports only a narrow slice of findings, creating the appearance of high precision while hiding missed weaknesses.
2. Escalation happens when the tool floods developers with noisy findings, causing teams to ignore future alerts and weakening the control environment.
3. Impact is realised when real vulnerabilities, including security flaws tied to secrets or application access paths, remain undiscovered and unremediated.

NHI Mgmt Group analysis

F1 score is a governance control, not just a benchmark: scanner accuracy metrics shape whether developers trust AppSec output enough to act on it. A metric that rewards silence or noise rather than balanced detection creates governance debt in the backlog. For programmes that touch secrets, service credentials, and other NHI-adjacent application risks, the control question is whether the tool produces decisions that developers can sustain.

False positive reduction is not the same as security improvement: a lower false positive rate can simply mean the vendor has hidden more real issues. That matters because operational teams rarely inspect the numerator and denominator closely enough to see the tradeoff. The result is a programme that appears efficient while quietly losing coverage, which is why balanced measurement should sit alongside NIST SP 800-53 Rev 5 Security and Privacy Controls and CIS Controls v8 thinking.

Named concept, detection trust gap: when scanner output becomes too noisy or too selective, developers stop treating findings as actionable evidence. That trust gap is the real failure mode, because once users ignore the queue, the tool stops functioning as part of the control system. For identity-adjacent application security, the implication is that measurement quality is part of governance quality, not an optional reporting layer.

Real production data is the only meaningful test of scanner claims: synthetic benchmarks can be useful for research, but they do not reflect the messiness of shipped software. The right question is whether the score comes from code teams actually run in production, with known classes of secrets, auth logic, and application flaws. Practitioners should treat dataset transparency as a minimum requirement for any procurement or tuning decision.

What this signals

Detection-trust gap: AppSec teams should expect scanner governance to become more measurement-driven as developers push back on noisy queues and low-confidence findings. The practical issue is not whether an engine can produce more alerts, but whether the programme can preserve trust in the alerts that matter. Where scanner output intersects with identity-related application controls, teams should align measurement with the NIST SP 800-53 Rev 5 Security and Privacy Controls view of accountable control operation.

A balanced evaluation standard also changes procurement conversations. Buyers will increasingly need to ask how the vendor measured performance, on what dataset, and whether precision, recall, and F1 were all disclosed. For teams managing secrets, auth logic, and code paths that influence access, that evidence is as important as feature claims.

The next maturity step is to connect scanner quality to operational outcomes. If developers stop trusting findings, the tool has already ceased to behave like a control. That makes measurement transparency part of software risk governance, not a reporting nicety.

For practitioners

• Require F1 score disclosure Ask every scanner vendor for an F1 score measured on real production code, not only on synthetic benchmarks or curated demo projects.
• Separate precision and recall in evaluation Insist on precision and recall being reported together so one metric cannot conceal weakness in the other.
• Test against your own codebase Run the scanner on representative production repositories that include real secrets, auth patterns, and application logic that mirror your environment.
• Review the dataset before the score Reject any benchmark claim that does not name the dataset, because the score is only meaningful if the code reflects your risk profile.
• Monitor developer trust in findings Track whether developers increasingly dismiss scanner output, since that is often the first sign that false positives are eroding control effectiveness.

Key takeaways

• AppSec scanner quality cannot be judged by precision or recall alone because each can hide the exact failure the other exposes.
• The most meaningful benchmark is F1 on real production code, because synthetic datasets can overstate how well a scanner will behave in practice.
• Teams should treat measurement transparency as a governance requirement, especially when scanner output influences secrets risk and application trust.

Key terms

• F1 Score: A combined metric that balances precision and recall in one number. In security scanning, it shows whether a tool is both finding real issues and avoiding excessive noise, which makes it more useful than either metric alone when comparing performance on real code.
• Precision: The share of reported findings that are actually real weaknesses. High precision means fewer false positives, but it does not guarantee that the scanner is finding all of the important problems, so it must be read alongside recall to understand practical effectiveness.
• Recall: The share of real weaknesses that the scanner successfully identifies. High recall reduces the chance of missed vulnerabilities, but if it is achieved by over-reporting, teams can become overwhelmed and stop trusting the results.
• False Positive: A false positive is a scanner result that looks like a secret but is not actually sensitive. In secret governance, false positives matter because they consume analyst time, weaken trust in alerts, and can delay response to the findings that truly change exposure and access risk.

What's in the full article

Checkmarx's full analysis covers the operational detail this post intentionally leaves for the source:

• Benchmark methodology across seven production codebases and how the test set was assembled
• The breakdown of precision, recall, and F1 for the hybrid SAST approach versus the LLM reviewer
• How the classification layer changes which findings reach analysts in practice
• The vendor's own guidance on using F1 when comparing scanner candidates

👉 Checkmarx's full post covers the benchmark methodology, metric breakdown, and scanner comparison details.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle fundamentals. It gives practitioners a shared control language for programmes that need stronger access, credential, and governance discipline.