Because the same SQL result can be judged differently under exact match, fact-based matching, or human review. A stricter rule may mark an equivalent answer wrong, while a looser rule may accept extra columns or reordered rows. The number is only meaningful if the rule is fixed in advance and applied consistently.
Why the score changes when the rubric changes
Text-to-SQL evaluation is not a single truth function. A model can generate SQL that returns the right result set, yet still fail under a stricter exact-match rubric, or pass under a more permissive result-based rubric. The score changes because the rubric decides what “correct” means, and different rubrics reward different kinds of equivalence, error tolerance, and formatting.
That means the metric is partly a statement about the test, not just the model. If the rubric treats reordered rows, extra columns, or semantically equivalent joins as acceptable, the measured score rises. If it demands exact structure or exact string correspondence, the same output can be counted as wrong even when the business answer is effectively the same.
Changing the rubric also changes what kinds of failures are visible. Exact-match scoring exposes syntax and formulation differences, while result-based or human review can expose whether the query actually answers the question, whether it is robust to schema quirks, and whether it hides a subtle logic error behind a plausible output. The evaluation method therefore shapes the headline number as much as the model does.
What each rubric is really measuring
Exact match is a structural test: it is useful when you need reproducibility, but it can penalise harmless variation. Fact-based matching is closer to outcome equivalence: it asks whether the returned data supports the same factual conclusion, even if the SQL differs. Human review is broader still, because it can judge intent, edge cases, and whether the query remains faithful to the prompt under realistic usage.
These measures are not interchangeable. A score under one rubric may reflect determinism, not usefulness; a score under another may reflect practical adequacy, not canonical SQL form. For that reason, text-to-SQL benchmarks should always report the rubric alongside the score, and readers should avoid comparing numbers across papers or systems unless the evaluation rule is the same.
That also explains why small rubric edits can produce large score swings. A rule that allows partial credit, ignores ordering, or accepts equivalent projections expands the set of “correct” answers. A rule that enforces exact clause structure narrows it. The score moves because the acceptance boundary moved.
How to interpret the number without over-reading it
The right question is not only “What score did the system get?” but also “What kind of correctness does this score represent?” If the benchmark is used to choose a production system, the rubric should resemble the operational standard you care about, not just the easiest test to pass. If the goal is research comparison, the rubric must stay fixed so that changes in the score reflect model progress rather than evaluation drift.
When two systems are close together, the rubric often matters more than the gap between them. One system may look better under exact match because it mirrors the benchmark style, while another may be more reliable in actual use because it consistently returns the right answer under looser semantic evaluation. The observed ranking can reverse when the scoring lens changes.
Risk and Threat Considerations
Evaluation drift creates a governance risk for text-to-SQL teams because a score can appear to improve even when the underlying capability has not changed. If teams compare results across mixed rubrics, they can overstate progress, miss regressions, or select the wrong system for deployment.
Failure mechanism: The benchmark acceptance rule changes, so the measured outcome shifts without a corresponding change in query quality. A permissive rubric can hide brittle SQL that only looks correct, while an overly strict rubric can understate practical performance and discourage useful models.
Impact: Teams can make misleading procurement, tuning, or release decisions, and published results become hard to compare over time or across studies.
Practitioner Guidance
What to verify: Lock the rubric before comparing systems, and record whether the score represents exact structure, result equivalence, or human judgement. If the rubric changes, treat the new number as a different metric, not a direct continuation of the old one.
What good looks like: Reports state the evaluation rule, the accepted equivalence class, and any tolerance for reordered rows, extra columns, aliases, or equivalent joins. That makes the score interpretable and reduces the chance of accidental apples-to-oranges comparisons.
Practitioner takeaway: In text-to-SQL, the score is only as meaningful as the rubric behind it, so consistency in evaluation matters as much as model quality.