Fact-level scoring evaluates leakage by individual security-relevant facts instead of by general similarity to the original text. This approach is better for agent security because an attacker can learn something valuable without reproducing the prompt verbatim.
What Fact-Level Scoring Measures
Fact-level scoring judges leakage by the specific security-relevant facts an output reveals, rather than by how closely it copies the source text. That makes it a better fit for agent security analysis, because a prompt can be meaningfully exposed even when it is not reproduced verbatim.
Why Fact-Level Scoring Is Better Than Similarity-Only Checks
Similarity-only approaches miss paraphrases, partial disclosures, and recomposed answers that preserve the sensitive substance while changing the wording. Fact-level scoring focuses on the informational units that matter, so it can detect when an attacker has extracted policy details, system instructions, tool access patterns, or other actionable knowledge without triggering a text-overlap test.
This distinction matters most when the secret or prompt is valuable in fragments. A model output can reveal one critical constraint, identifier, dependency, or operational rule that is enough to aid follow-on abuse, even if the surrounding language looks harmless.
How Fact-Level Scoring Is Used in Evaluation
In practice, evaluators define the facts they care about, then score whether those facts appear in the model output with enough fidelity to be considered leaked. The method is usually more expressive than n-gram similarity, because it can account for paraphrase, abstraction, and partial reconstruction of the source material.
That makes it useful for red teaming, benchmark design, and regression testing of prompt leakage defenses. It also helps teams compare systems more fairly, because a model that “sounds different” is not necessarily safer if it still exposes the same operational content.
For adjacent context on how scoring and prioritisation differ from exploit likelihood or vulnerability severity, compare FIRST CVSS and FIRST EPSS, which measure different security questions altogether.
Where Fact-Level Scoring Fits in Agent Security
Fact-level scoring is especially useful for agents because leakage can expose instructions, constraints, secrets-handling patterns, tool names, routing logic, or escalation paths that are individually useful to an adversary. The security question is not just whether the prompt was copied, but whether the model disclosed enough of the underlying operational facts to change an attacker’s capabilities.
That is why the method supports a more realistic assessment of harm: it measures whether the model has preserved the sensitive meaning, not merely the original wording. For teams building evaluation pipelines, that usually makes the metric a better proxy for practical exposure than similarity thresholds alone.
For a broader reference point on adjacent vulnerability intelligence workflows, the NIST National Vulnerability Database shows how security ecosystems often track materially important facts, not just surface resemblance.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST AI RMF, NIST CSF 2.0 and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | Govern | Fact-level scoring supports AI risk measurement and evaluation of harmful information disclosure. |
| Recommendation — Use AI RMF measurement and monitoring to test whether evaluations capture meaningful disclosure, not just text similarity. | ||
| NIST CSF 2.0 | DE.CM-01 — Monitoring for Anomalies and Events | Leakage scoring is a detection-oriented control that helps observe when sensitive facts are exposed. |
| Recommendation — Tune monitoring to flag outputs that reveal protected facts even when wording differs from the source. | ||
| NIST SP 800-53 Rev 5 | AU-6 — Audit Record Review, Analysis, and Reporting | Fact-level scoring functions as analytic review of disclosures and supports post-event assessment. |
| Recommendation — Review sampled outputs for disclosed facts and report leakage patterns that bypass similarity checks. | ||