Text-to-SQL benchmarks: what changes when the rule changes?

TL;DR: Databricks Genie beats Snowflake Cortex Analyst on all three corpora, according to Agentic Fabriq, but the lead on Spider 1.0 ranges from 7.2 points under exact match to 0.4 under got-the-facts, while refusal rates and schema-loading choices explain part of the spread. The real buyer risk is treating a benchmark number as stable when the grading rule, corpus and harness conditions can change the answer.

Editorial analysis by NHI Mgmt Group, based on content published by Agentic Fabriq: “Snowflake Cortex Analyst vs Databricks Genie: Accuracy, With the Rule Attached”.

By the numbers:

• Databricks Genie scored 37.4 / 53.1 on BIRD mini-dev under exact match / got-the-facts.
• Snowflake Cortex Analyst scored 56.4 / 77.1 on Spider 1.0 under exact match / got-the-facts.
• On Spider 2.0, Cortex Analyst had 27 refusals out of 135 answers.

Key questions

Q: How should teams evaluate text-to-SQL tools without relying on misleading leaderboards?

A: Start by matching the benchmark to your environment, then choose the scoring rule that reflects how your users will judge a correct answer.

Q: Why do text-to-SQL benchmarks produce different winners under different rules?

A: Because exact match and fact containment reward different things.

Q: What are the signs that a text-to-SQL evaluation is too dependent on metadata quality?

A: High refusal rates, sensitivity to missing join relationships, and large swings between corpora usually indicate that the setup is shaping the score.

Practitioner guidance

• Fix the grading rule before you compare vendors Define whether your use case needs exact-match equivalence, fact containment or another metric, then apply the same rule to every candidate.
• Test on your own warehouse and question set Run a small set of real analyst questions against both systems, using your own schema, joins and semantic annotations so the result reflects your environment.
• Track refusals as a separate KPI Record how often each system refuses, why it refuses and whether the refusal is caused by missing metadata, scope limits or an actual inability to answer.

Bottom line: The main risk is not only model quality but benchmark instability, because the same system can look materially better or worse when the scoring rule changes.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →