TL;DR: Text-to-SQL leaderboard rankings shift when the benchmark corpus and grading rule change, even for the same systems and questions, according to Agentic Fabriq: on Spider 1.0, Databricks Genie led Snowflake Cortex Analyst by 7.2 points under exact match but only 0.4 points under got-the-facts. The practical lesson is that benchmark scores are filters, not procurement decisions, because answer shape and annotation error can reorder the board.
At a glance
What this is: NHI Mgmt Group's analysis, drawing on Agentic Fabriq's research: BIRD and Spider text-to-SQL rankings disagree because leaderboard order depends on the benchmark corpus, the grading rule, and the quality of the answer key.
Why it matters: IAM and security teams should treat any published performance score as context, not proof, because evaluation design can change rankings without any change in the underlying system.
Context
Text-to-SQL evaluation is not a single measurement problem. Different benchmarks test different database shapes, different question styles, and different scoring rules, so a score is only meaningful when those conditions are stated together.
That matters for identity and governance programmes because vendor selection, internal model approval, and control validation can all be distorted by a score that looks precise but is actually benchmark-specific. The question is not which system won in the abstract, but which system fits your workload and your grading rule.
Key questions
Q: How should teams evaluate text-to-SQL tools without relying on misleading leaderboards?
A: Start by matching the benchmark to your environment, then choose the scoring rule that reflects how your users will judge a correct answer. A leaderboard is only useful as a screening signal. Final selection should use your own schema, your own query patterns, and a grader that reflects business correctness rather than benchmark convenience.
Q: Why do two text-to-SQL benchmarks rank the same system differently?
A: Because the benchmark corpus, the answer key, and the grading rule all influence the result. If one test rewards exact shape and another rewards partial containment, the same output can move from wrong to right without the model changing. That is why cross-benchmark rankings are not interchangeable.
Q: What should practitioners check before trusting a published AI benchmark score?
A: Check the corpus, the grading rule, the version of the dataset, and whether the reference answers were audited for errors. A score without those details may describe the test more than the system. If the evaluation conditions differ from your environment, treat the number as directional only.
Q: When is a small leaderboard gap not meaningful for procurement?
A: When the benchmark has known annotation errors or when the scoring rule is permissive enough that small differences do not map to real operational differences. In those cases, a narrow margin can sit inside measurement noise. Use the score to shortlist candidates, not to declare a winner.
Technical breakdown
Why grading rules reorder text-to-SQL results
A text-to-SQL grader converts an output table into a pass or fail, and small differences in that conversion change the final ranking. Exact set match rewards answers that reproduce the reference shape closely, execution accuracy checks whether the query returns the right result, and containment or got-the-facts accepts any output that contains the reference values. These rules do not measure the same thing, so they should not be expected to produce the same leaderboard order. In practice, answer shape can matter as much as semantic correctness when the evaluator is strict.
Practical implication: Choose the grading rule that matches how your users will judge a correct answer, not the rule that makes the leaderboard look cleaner.
Why benchmark corpora are not interchangeable
Spider 1.0, BIRD, and Spider 2.0 were built for different jobs. Spider 1.0 tests schema generalisation across many small databases, BIRD adds dirty data, external knowledge, and efficiency pressure, and Spider 2.0 targets enterprise workflows with large schemas and long multi-query solutions. A system that is strong on one corpus may fail on another because the data shape, question style, and expected reasoning steps are different. Comparing scores across those corpora as if they were the same test turns evaluation into a false equivalence.
Practical implication: Map each benchmark to the production environment it actually resembles before you use the score for selection or risk review.
Why annotation errors create false confidence
A leaderboard is only as reliable as its answer key. If the reference SQL, database schema, or question wording is wrong, a correct system can be penalised and an inferior one can be rewarded. That is why corrected samples can move rankings several positions without any model change. The problem is not just noise, but systematic evaluation debt: the benchmark itself carries unresolved mistakes that travel into procurement, benchmarking, and progress claims.
Practical implication: Audit a benchmark’s reference quality before treating small score gaps as meaningful evidence of superiority.
Threat narrative
Attacker objective: The practical objective is not theft but misclassification: to make a system appear more capable than it will be in the buyer's environment.
- Entry occurs when a buyer treats a leaderboard score as a universal signal rather than a benchmark-specific measurement.
- Credential access is not the issue here; the comparable weakness is evaluation key access, where the hidden assumptions inside the grading rule determine the result.
- Impact appears when procurement or approval decisions rely on a ranking that does not transfer to the target warehouse or workflow.
Breaches seen in the wild
- SAP SQL Anywhere Monitor hard-coded credentials (CVE-2025-42890): Hard-coded credentials in SAP SQL Anywhere Monitor scored CVSS 10.0 (CVE-2025-42890); SAP removed the tool, and no exploitation was seen.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Benchmark-driven ranking is not a property of the system, it is a property of the test. The same model can look stronger or weaker once the corpus and grader change, which means leaderboard order is an artefact of measurement design as much as model quality. For identity and security teams, that matters because assurance processes often inherit the same false certainty from scored comparisons. The right conclusion is to validate the measurement model before trusting the result.
Answer shape is a governance issue, not a minor scoring detail. If one evaluator tolerates extra columns, alternate ordering, or looser containment while another insists on exact matches, they are rewarding different operational behaviours. That difference is large enough to reverse rankings, which means procurement teams need to know which behaviour they are optimising for. The practical conclusion is to align the score with the user outcome, not with the easiest metric to publish.
Evaluation debt accumulates when reference data is treated as ground truth without audit. The article's strongest evidence is not just that benchmarks disagree, but that annotation errors can move rankings by multiple positions after correction. That is a familiar governance failure pattern in any control environment: people trust the reported number more than the quality of the method behind it. The practical conclusion is to inspect the benchmark as carefully as the system being measured.
Text-to-SQL procurement needs a context-aware benchmark policy, not a single winner's list. A score can still be useful as a filter, but only when the corpus resembles the production setting and the grading rule reflects the business definition of correctness. In practice, that means the board should inform shortlisting, not final selection. The practical conclusion is to decide with your own schema and your own success criteria.
What this signals
Benchmark fidelity is now a procurement control. If a team cannot state which corpus and grading rule produced a score, the number should not drive a purchasing or approval decision. The safest operating model is to treat benchmark evidence as one input, then validate the finalist against live schemas and business-style queries.
The deeper lesson is that measurement design now sits inside the control plane. For any AI system that will touch enterprise data, teams should ask whether the published score reflects the actual failure mode they care about, or merely a convenient test condition.
For practitioners
- Define the production-like evaluation rule Choose exact match, execution accuracy, or containment based on how your warehouse users will judge correctness, then document that choice before comparing vendors.
- Compare only like-for-like corpora Require benchmark context, including database shape, question type, and workload complexity, before treating scores as comparable across systems.
- Audit the reference key before buying Check whether the benchmark has known annotation errors, ambiguous questions, or schema issues that could distort the ranking you are using.
- Shortlist before you standardise Use published scores to narrow the field, then run the finalists against your own schema and business questions with the rule your users care about.
Key takeaways
- Text-to-SQL leaderboard order can change without any model change when the corpus or grading rule changes.
- Annotation errors in the reference set can move rankings enough to make small score gaps meaningless.
- The practical response is to use published scores for screening and then test finalists against your own schema and success criteria.
Related NHI Mgmt Group resources: SQL Injection Prevention Guide
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST CSF 2.0, NIST AI RMF and NIST SP 800-53 Rev 5 set the technical controls, while ISO/IEC 27001:2022 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | GV.OV-01 — Oversight of cybersecurity risk management | Benchmark selection and score interpretation are governance decisions that require oversight. |
| Recommendation — Apply governance oversight to ensure benchmark scores are interpreted within their intended measurement limits. | ||
| NIST AI RMF | MEASURE — Measure AI risk and performance | The article is fundamentally about how measurement design changes perceived AI performance. |
| Recommendation — Measure AI systems with the same corpus and grading rules you will use for real decisions. | ||
| NIST SP 800-53 Rev 5 | CA-2 — Security Assessments | Comparable evaluation of AI tools needs structured assessment, not anecdotal score reading. |
| Recommendation — Use security assessment discipline to validate benchmark claims before procurement or approval. | ||
| ISO/IEC 27001:2022 | A.5.36 — Compliance with policies, rules and standards for information security | Published scores must be used according to a defined internal policy for evidence and vendor review. |
| Recommendation — Set policy for how benchmark results can be used in assurance and sourcing decisions. | ||
Key terms
- Benchmark Corpus: The benchmark corpus is the set of questions, databases, and reference answers used to test a system. In text-to-SQL evaluation, the corpus determines what kind of reasoning is being measured, so corpus choice can change rankings even when the model is unchanged.
- Grading Rule: The grading rule is the logic used to decide whether a system output counts as correct. It can compare exact result sets, execution output, or containment of reference values, and each rule rewards a different answer shape and creates different leaderboard outcomes.
- Answer Shape: Answer shape is the structure of the returned result, including columns, ordering, and whether extra values are present. A system can be semantically correct but still fail under a strict evaluator if its answer shape does not match the reference closely enough.
- Evaluation Debt: Evaluation debt is the gap between the tests a team has and the behaviours the product now exhibits in production. It grows when datasets, thresholds, and judges are not updated as the system changes, leaving organisations with scores that no longer reflect real risk.
Deepen your knowledge
NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is designed for practitioners who need to connect governance decisions to real operational control.
Published by the NHIMG editorial team on October 22, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org