Why do text-to-SQL benchmarks produce different winners under different rules?

Why evaluation rules change the result

Text-to-SQL benchmarks reward different qualities depending on the scoring rule. Exact match treats the SQL or output as a shape-sensitive artifact, while fact containment cares about whether the returned result preserves the right information. That means a model can look weaker under one rule and stronger under another, even when the underlying answer quality is similar.

The practical implication is that benchmark winners are partly a function of what the benchmark is optimising for. Systems that generate faithful SQL structure tend to do better on exact match, while systems that recover the right rows or facts in a less canonical form can do better on looser evaluation.

Why exact match and fact containment diverge

Exact match is unforgiving about syntax, ordering, aliases, joins, and other structural choices. If the query is semantically close but not identical, the score can still be zero. Fact containment is more tolerant because it asks whether the answer includes the correct facts, even if the expression of those facts differs from the reference.

This creates a real gap between correctness as a database engineer sees it and correctness as a benchmark sees it. Two outputs can be operationally similar but benchmarked differently because one is an exact textual or logical match and the other is only partially aligned to the reference shape.

For benchmarking in adjacent software domains, hard structure checks are often paired with control baselines so teams can understand where a rule is measuring formatting discipline versus actual task success. That kind of evaluation design is part of why CIS Benchmarks are useful as a comparison point: they remind teams that scoring criteria need to reflect the control objective, not just surface similarity.

What benchmark buyers should compare, not just the score

When a buyer chooses a text-to-SQL system, the more important question is what failure mode matters most. If the application requires exact downstream execution, strict structural accuracy matters. If the workflow mainly serves analysts who need the right facts quickly, looser fact-based success may be more useful.

That trade-off becomes material when teams compare models across datasets that use different annotation styles or SQL canonicalisation rules. A system can appear to regress simply because the evaluation got stricter, not because it became less useful in practice.

Security and governance teams face a similar problem when they compare controls by outcome rather than by metric definition. A benchmark or control only has meaning if the scoring rule matches the decision being made, so teams should document what “success” means before treating one winner as universally better than another.

Risk and Threat Considerations

Evaluation drift is the main risk: if teams assume one benchmark winner is objectively superior, they may choose a system that is optimised for the wrong notion of correctness. In production, that can lead to silent analytic errors, brittle integrations, or false confidence in an assistant that only looks good under one scoring rule.

Failure mechanism: Different rules reward different output properties, so a model can overfit to exact structure, over-deliver on partial facts, or game the benchmark without being robust across real usage patterns.

Impact: Procurement decisions, model comparisons, and regression testing can all be misleading, especially when the benchmark is used as a proxy for user value rather than a narrow lab metric.

Practitioner Guidance

What to verify: Check whether the benchmark score reflects the actual user task. If users need machine-executable SQL, exact match is more relevant; if they need answer fidelity for human review, fact containment may be the better primary signal.

Decision rule: Treat a model as “better” only when it wins under the evaluation rule that matches the production requirement. If the rules differ, document the trade-off instead of collapsing them into one headline winner.

Practitioner takeaway: The important lesson is not that one model is objectively best, but that benchmark design determines what kind of correctness gets rewarded, so the winner depends on the decision you are actually trying to make.

Related resources from NHI Mgmt Group

• What should teams do when an AI chatbot needs to query data under different permission rules?
• How should teams structure prompts for text-to-SQL systems to improve accuracy across different databases?
• How should security teams interpret ransomware benchmarks that use different counting rules?
• Why do SAST and SCA need different triage rules?