Why do recall and precision both matter in SAST evaluation?

How recall and precision work together in SAST

Static analysis is only useful when teams trust both what it catches and what it leaves out. Recall measures coverage of real defects, while precision measures the share of alerts that are actually worth investigation. Evaluating only one can produce the wrong operational outcome: either a quiet tool that misses risk, or a noisy tool that overwhelms reviewers.

These two measures also describe different failure modes. Recall tells you whether the scanner is finding enough of the issues you care about in the codebase you run it against. Precision tells you whether the findings are discriminating enough to support remediation work without wasting engineering time. A good SAST program needs both because review capacity is finite and missed vulnerabilities have real cost.

In practice, recall and precision should be read as a pair rather than as competing goals. Improving one often degrades the other unless the rule set, tuning, code patterns, and triage process are handled carefully. That is why evaluation should reflect the actual operating environment, not just a synthetic benchmark or a vendor demonstration.

Why a single metric gives a misleading picture

High recall with weak precision usually means the scanner is broad but not selective. Teams may feel well covered, yet reviewers can become desensitised by repeated false positives, which slows down triage and can cause real alerts to be ignored. This is especially costly when security findings compete with feature work and deadline pressure.

High precision with weak recall creates the opposite problem. The report looks clean and efficient, but important flaws remain undiscovered because the scanner is too narrow, too conservative, or poorly aligned to the code patterns in use. That can create a false sense of security, which is often more dangerous than a noisy but honest tool.

A practical evaluation should therefore ask what trade-off the team is actually accepting. If the scanner is used to gate builds, precision matters because false alarms block delivery. If it is used to discover latent defects before release, recall matters because missed issues defeat the purpose of scanning. Most mature programmes need both behaviours across different stages of the pipeline.

What good SAST evaluation looks like in practice

The right test is not “does the tool find issues?”, but “does it find enough real issues with a review burden the team can sustain?” That means looking at defect classes, language coverage, rule quality, suppression behaviour, and how often findings are actionable. It also means checking whether the scanner’s results are stable enough to support trending and operational decisions over time.

Evaluation should include representative code, common libraries, project-specific patterns, and the way the team actually builds and deploys software. A scanner that performs well on contrived examples may struggle with real application structure, framework conventions, or custom wrappers around risky functions. This is where tuning and human review discipline matter as much as the underlying detection engine.

For a useful baseline, compare results against a hand-reviewed sample or known vulnerability set, then inspect not just counts but the distribution of true positives, false positives, and misses. Teams that care about both coverage and workload often pair SAST with other verification methods so the scanner is one signal among several, not the sole source of truth.

Risk and Threat Considerations

When recall is too low, exploitable code can ship because the scanner never surfaced it. When precision is too low, teams spend security effort on low-value noise and may underinvest in the findings that matter most. The operational risk is not just inefficiency, it is misallocated attention at exactly the point where secure coding feedback should be most actionable.

Failure mechanism: Poor rule coverage, brittle pattern matching, or weak tuning reduces recall, while overly broad signatures, ambiguous dataflow interpretation, or noisy rule packs reduce precision. Either failure mode distorts the triage queue and weakens confidence in the tool.

Impact: Missed defects increase the chance of shipping vulnerable code, while excessive false positives increase review fatigue, delay releases, and can cause teams to ignore scanner output. At scale, this can turn SAST into a compliance activity rather than a meaningful control.

Practitioner Guidance

What to verify: Test the scanner against code that reflects your real frameworks, libraries, and custom patterns, then inspect both missed findings and the proportion of alerts your team can confirm as real. A vendor score alone is not enough; the operational question is whether security and engineering can sustain the output.

Decision rule: If the tool finds many real issues but overwhelms reviewers, tune for precision before expanding rollout. If the findings look clean but known defect classes are absent, prioritise recall and challenge rule coverage before trusting the result.

Practitioner takeaway: The best SAST program is not the one with the highest score on one metric, but the one whose false negatives and false positives are both low enough to support steady, credible remediation.

Related resources from NHI Mgmt Group

• Should organisations prioritise recall or precision when choosing a SAST platform?
• What is the difference between precision and recall in AI evaluation?
• What is the difference between precision and recall in SAST buying decisions?
• Why does data-shape fit matter so much in policy evaluation systems?