Join our Newsletter — 33% off our NHI Course

Did BIRD and Spider rankings disagree because of benchmark rules?

 

(@nhi-mgmt-group)
Member Moderator
Joined: 2 years ago
Posts: 20225
Topic starter  

TL;DR: Benchmark corpus and grading rule changes can reorder text-to-SQL leaderboards for the same systems, with Agentic Fabriq showing Databricks Genie ahead of Snowflake Cortex Analyst by 7.2 points under exact match but only 0.4 under got-the-facts. Text-to-SQL evaluation needs schema-matched tests and business-relevant scoring before published gaps guide selection.

Editorial analysis by NHI Mgmt Group, based on content published by Agentic Fabriq: “Why Do BIRD and Spider Rankings Disagree?”.

Key questions

Q: Why do two text-to-SQL benchmarks rank the same system differently?

A: Because the benchmark corpus, the answer key, and the grading rule all influence the result.

Q: What should practitioners check before trusting a published AI benchmark score?

A: Check the corpus, the grading rule, the version of the dataset, and whether the reference answers were audited for errors.

Q: How should teams evaluate text-to-SQL tools without relying on misleading leaderboards?

A: Start by matching the benchmark to your environment, then choose the scoring rule that reflects how your users will judge a correct answer.

Practitioner guidance

  • Define the production-like evaluation rule Choose exact match, execution accuracy, or containment based on how your warehouse users will judge correctness, then document that choice before comparing vendors.
  • Compare only like-for-like corpora Require benchmark context, including database shape, question type, and workload complexity, before treating scores as comparable across systems.
  • Audit the reference key before buying Check whether the benchmark has known annotation errors, ambiguous questions, or schema issues that could distort the ranking you are using.

Bottom line: Text-to-SQL leaderboard order can change without any model change when the corpus or grading rule changes.

Glossary terms: Benchmark Corpus, Grading Rule, Answer Shape, Evaluation Debt

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →


This topic was modified 1 week ago by NHI Mgmt Group

   
Quote
(@mr-nhi)
Member Moderator
Joined: 6 months ago
Posts: 20715
 

Benchmark-driven ranking is not a property of the system, it is a property of the test. The same model can look stronger or weaker once the corpus and grader change, which means leaderboard order is an artefact of measurement design as much as model quality. For identity and security teams, that matters because assurance processes often inherit the same false certainty from scored comparisons. The right conclusion is to validate the measurement model before trusting the result.

A question worth separating out:

Q: When is a small leaderboard gap not meaningful for procurement?

A: When the benchmark has known annotation errors or when the scoring rule is permissive enough that small differences do not map to real operational differences. In those cases, a narrow margin can sit inside measurement noise. Use the score to shortlist candidates, not to declare a winner.

👉 Read our full editorial: Text-to-SQL leaderboards diverge because benchmarks reward different things


This post was modified 1 week ago by NHI Mgmt Group

   
ReplyQuote
Share:

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.