Why do BIRD and Spider rankings disagree on the same systems?

TL;DR: Text-to-SQL leaderboard rankings shift when the benchmark corpus and grading rule change, even for the same systems and questions, according to Agentic Fabriq: on Spider 1.0, Databricks Genie led Snowflake Cortex Analyst by 7.2 points under exact match but only 0.4 points under got-the-facts. The practical lesson is that benchmark scores are filters, not procurement decisions, because answer shape and annotation error can reorder the board.

Editorial analysis by NHI Mgmt Group, based on content published by Agentic Fabriq: “Why Do BIRD and Spider Rankings Disagree?”.

Key questions

Q: How should teams evaluate text-to-SQL tools without relying on misleading leaderboards?

A: Start by matching the benchmark to your environment, then choose the scoring rule that reflects how your users will judge a correct answer.

Q: Why do two text-to-SQL benchmarks rank the same system differently?

A: Because the benchmark corpus, the answer key, and the grading rule all influence the result.

Q: What should practitioners check before trusting a published AI benchmark score?

A: Check the corpus, the grading rule, the version of the dataset, and whether the reference answers were audited for errors.

Practitioner guidance

• Define the production-like evaluation rule Choose exact match, execution accuracy, or containment based on how your warehouse users will judge correctness, then document that choice before comparing vendors.
• Compare only like-for-like corpora Require benchmark context, including database shape, question type, and workload complexity, before treating scores as comparable across systems.
• Audit the reference key before buying Check whether the benchmark has known annotation errors, ambiguous questions, or schema issues that could distort the ranking you are using.

Bottom line: Text-to-SQL leaderboard order can change without any model change when the corpus or grading rule changes.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →