What is the difference between factuality scores and Levenshtein scores in model evaluation?

Factuality measures whether the extracted content is semantically correct, while Levenshtein distance measures how closely the output text matches a reference string. They can move differently. A model may look acceptable on edit distance but still get key values wrong, so teams should use both metrics when judging extraction quality.

Why factuality and string similarity answer different questions

Factuality scores and Levenshtein scores are often discussed together because both can be used to judge model outputs, but they are not measuring the same failure mode. Factuality asks whether the extracted value is correct in meaning, while Levenshtein asks how much the output text differs from a reference string. That distinction matters most when the task is extraction, where a near match can still be wrong in a way that changes the result.

For example, a system can produce text that is close enough to a reference to earn a strong edit-distance score while still returning the wrong date, amount, name, or identifier. That is why string similarity is useful as a formatting or recall signal, but not as a substitute for semantic correctness. Teams that confuse the two tend to overestimate quality when outputs “look close” rather than when they are actually right. In practice, many teams discover this only after a downstream report, lookup, or workflow has already accepted a superficially close but incorrect value.

When a process depends on exact fields, factuality is the more decision-relevant measure because it tracks whether the model preserved meaning, not just character sequence. A strict edit-distance score can still reward outputs that are operationally unusable.

How the two metrics behave in real evaluation pipelines

Levenshtein distance is a character-level comparison. It counts insertions, deletions, and substitutions needed to turn one string into another. That makes it useful when the task has a fixed expected form, such as normalised codes, short labels, or tightly constrained extraction fields. It is also easy to compute and easy to explain, which is why it often appears in early evaluation harnesses.

Factuality is broader and usually requires a judgment about whether the extracted content preserves the underlying fact, even if the wording differs. In practice, that may mean comparing dates after normalisation, matching entities despite punctuation or casing changes, or checking whether a value is the correct one rather than merely a similar one. For an extraction system, factuality better captures whether the model understood the source and returned the right answer.

The two scores diverge whenever the “right” answer can be expressed in multiple valid forms, or when the text can be close but still materially wrong. That divergence is especially visible in fields like addresses, identifiers, measurements, legal text, and structured records. A high Levenshtein score may indicate that the model copied the shape of the answer well, but it does not prove the answer is true. Conversely, a lower edit-distance score may still be acceptable if the output is semantically correct after normalisation.

Use Levenshtein when surface form is part of the requirement.
Use factuality when correctness of meaning is the real objective.
Normalise formats before comparing either score when spacing, punctuation, or casing are irrelevant.
Judge task success against the field semantics, not against textual resemblance alone.

The guidance breaks down when the reference itself is ambiguous or when multiple outputs are equally valid and no normalisation rule has been defined.

Where the differences become most obvious

Tighter evaluation often increases scoring complexity, so teams have to balance simplicity against the risk of rewarding the wrong behaviour. That tradeoff is most visible when outputs are short, highly structured, or used in downstream automation, because a tiny textual deviation can have a very different operational meaning from a tiny factual deviation.

One common edge case is partial correctness. A model may capture the right entity but miss a qualifier, making the output look close in Levenshtein terms while failing a factual review. Another is synonymy and paraphrase: the model may express the same fact with different wording, which can penalise edit distance without indicating a real error. A further complication is reference quality. If the gold answer is inconsistently formatted or incomplete, Levenshtein can understate performance and factuality judgments can become inconsistent across reviewers.

That is why there is no universal consensus that one metric should replace the other. The better practice is to choose the metric that matches the task definition, then use the second metric as a check for blind spots. For extraction-heavy evaluation, factuality usually carries more weight, while Levenshtein remains useful for spotting formatting drift and regression in exact-match style outputs.

When teams need a quick quality signal, they should ask whether the user would care more about being close to the reference string or being correct about the underlying fact.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

NIST AI RMF, NIST AI 600-1 and CIS Controls v8 set the technical controls, while ISO/IEC 42001:2023 define the regulatory obligations.

Framework	Control / Reference	Relevance
NIST AI RMF	MEASURE — Measure and Evaluate	Evaluation metrics must distinguish semantic quality from format similarity.
Recommendation — Measure semantic correctness and surface-form stability separately in your model evaluation plan.
ISO/IEC 42001:2023	9.1 — Monitoring, measurement, analysis and evaluation	This topic concerns selecting and interpreting AI evaluation measures.
Recommendation — Define evaluation measures that reflect the actual quality property you need to govern.
NIST AI 600-1	EVAL — Evaluation and Testing	Model outputs should be tested with metrics that match the task objective.
Recommendation — Use task-aligned evaluation tests instead of relying on a single proxy score.
CIS Controls v8	8 — Audit Log Management	Control selection aligns with preserving evidence and quality signals for automated systems.
Recommendation — Retain evaluation evidence and trend both metrics so regressions are visible over time.

Practitioner Guidance

What to prioritise: Treat factuality as the primary metric when the output will drive a decision, populate a record, or be consumed by another system. Use Levenshtein as a supporting metric for surface-form stability, not as the main definition of correctness.

What to verify: Check whether your evaluation set allows multiple valid phrasings, normalised formats, or equivalent representations. If it does, raw edit distance will overstate error unless you standardise the comparison rules first.

Decision rule: If a small text change would not change the business meaning, Levenshtein should not dominate the scorecard; if the exact text form matters, then it should.

Practitioner takeaway: The strongest evaluation setup separates semantic correctness from textual similarity, because those metrics catch different failures and prevent a model from being judged “good” for the wrong reason.