Pooled Recall

Pooled recall is the total number of unique vulnerabilities found across multiple runs of the same model. It captures the benefit of repetition when different passes surface different issues. This metric is useful when assessing search-oriented security workflows where one run alone does not represent full capability.

Expanded Definition

Pooled recall is a coverage metric used to describe how many unique vulnerabilities are discovered when results from multiple runs are combined. In practice, it is most relevant for repeatable security testing workflows where one pass may miss issues that another pass can surface. The metric helps distinguish raw detection performance from cumulative discovery performance, which is especially important when evaluating search-heavy processes such as code review assistants, vulnerability triage pipelines, and agentic scanning workflows.

Definitions vary across vendors and research contexts, but the core idea is stable: if repeated runs identify different findings, pooled recall measures the total distinct set recovered across those runs. That makes it different from single-run recall, which only reflects one execution. For security teams, the term is useful when comparing tools or prompts that produce probabilistic outputs, because it captures the value of repetition rather than treating each run as an isolated event. The most common misapplication is treating pooled recall as proof of accuracy, which occurs when teams ignore false positives, duplicate findings, or the fact that more runs can inflate apparent coverage without improving signal quality.

Examples and Use Cases

Implementing pooled recall rigorously often introduces a validation burden, requiring teams to weigh broader discovery against the cost of deduplication and manual confirmation.

• A red team runs the same vulnerability discovery workflow several times against a codebase, then pools the unique findings to estimate overall search coverage.
• A security engineering team compares two AI-assisted review prompts by measuring which one surfaces the larger combined set of distinct issues across repeated trials.
• An AppSec program uses pooled recall to evaluate whether an autonomous scanner is consistently missing the same classes of flaws or only finding them intermittently.
• A model governance team applies the metric alongside NIST Cybersecurity Framework 2.0 outcomes to understand whether a search process is producing reliable coverage for risk decisions.
• An evaluation lab pools results from multiple retrieval-augmented security test runs to compare search quality across different configurations, especially where output varies from run to run.

In these cases, pooled recall is most useful when the underlying task is exploratory rather than deterministic. It helps teams ask whether repetition is uncovering genuinely new weaknesses, not just re-reporting the same ones in different language. It is also helpful when assessing AI agents that can take multiple paths through a workflow, since a single execution may not reflect the agent’s full discovery potential.

Why It Matters for Security Teams

Pooled recall matters because security teams often need to know whether a workflow is broadly capable of finding problems, not merely whether it succeeds once. That distinction is critical in AI-assisted testing, NHI discovery, and other search-oriented tasks where autonomy and variability can change the result from one run to the next. Used well, the metric supports better benchmarking, more honest tool comparisons, and stronger decisions about whether a workflow is mature enough for operational use.

It also has governance value. If a team measures only single-run results, it can overestimate the reliability of a scanner, an agent, or a human-plus-AI review process. That creates blind spots in risk reporting and can leave vulnerable assets unexamined. Pooled recall does not replace precision, remediation prioritisation, or severity assessment, but it gives leaders a clearer picture of discovery coverage. For identity-linked systems, especially those involving NIST Cybersecurity Framework 2.0 governance practices, this matters when automated review is used to inspect credentials, secrets, or agent permissions at scale. Organisations typically encounter the limits of pooled discovery only after repeated scans still miss the same critical weakness, at which point the metric becomes operationally unavoidable to address.

Related resources from NHI Mgmt Group

• Why is syntax recall a weaker signal than architectural judgement?
• Why do pooled telecom identities increase operational risk?
• Why do precision and recall create blind spots in scanner selection?
• How do teams decide between ROC and precision-recall curves?