Got-the-facts is a looser scoring rule that accepts an answer when it contains the reference result, even if extra columns or formatting differences are present. It is useful when factual retrieval matters more than shape, but it can overstate practical query quality.
What Got-the-Facts Measures
Got-the-facts is a permissive evaluation rule, not a strict quality check. It treats an answer as acceptable when it returns the reference result, even if the shape, column order, or formatting differs from the expected output.
Why It Is Useful
This rule is helpful when the core concern is factual retrieval, where the most important question is whether the right answer appears at all. It can reduce false negatives in systems that compare answers across slightly different schemas or presentation styles.
That same leniency is also the reason it needs careful interpretation: a system can look successful on got-the-facts while still producing output that is awkward, inconsistent, or hard to use in practice.
Where It Fits in Evaluation
Got-the-facts sits closer to answer correctness than to formatting fidelity. It is a looser signal than exact-match scoring, so it is best used when the task values semantic accuracy over rigid output shape.
For teams measuring retrieval or question answering quality, it can be a useful companion metric, especially during early development or when the downstream consumer can tolerate structural variation.
What It Can Hide
The main limitation is that it may overstate practical performance. A system can “pass” because it surfaced the right fact, while still failing on usability, presentation consistency, or downstream automation requirements.
That means got-the-facts should not be the only success criterion when output shape matters. It is strongest as a tolerance rule for factual relevance, not as a substitute for more exact evaluation.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP ASVS, NIST CSF 2.0 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP ASVS | V16 — Security Logging and Error Handling | Evaluation outputs need consistent, reviewable handling when accuracy and format differ. |
| Recommendation — Verify logging and error handling so evaluation results remain traceable when outputs vary in shape. | ||
| NIST CSF 2.0 | GV.OV-01 — Outcomes and Metrics Evaluation | The term concerns how a quality metric is interpreted and used in assessment. |
| Recommendation — Define metric thresholds so factual success does not mask format or usability failures. | ||
| CIS Controls v8 | CIS-8 — Audit Log Management | Quality evaluation benefits from recorded results that can be reviewed and compared over time. |
| Recommendation — Retain evaluation records so you can inspect why an answer passed or failed. | ||
Related resources from NHI Mgmt Group
- Why do static LLM tests fail when an answer depends on current facts?
- Why does understanding the exposed data matter more than knowing only how an attacker got in?
- How do security and platform teams decide which user facts should be retained in AI memory?
- Why do stale dependency facts cause AI incident tools to make bad decisions?