Topic starter
25/10/2026 11:00 am
TL;DR: Databricks Genie leads Snowflake Cortex Analyst on all three corpora, with Agentic Fabriq tying the spread to the grading rule, refusal rates, and schema-loading choices. Text-to-SQL teams should treat benchmark scores as setup-dependent unless the same corpus and scorer are kept fixed.
Editorial analysis by NHI Mgmt Group, based on content published by Agentic Fabriq: “Snowflake Cortex Analyst vs Databricks Genie: Accuracy, With the Rule Attached”.
By the numbers:
- Databricks Genie scored 37.4 / 53.1 on BIRD mini-dev under exact match / got-the-facts.
- Snowflake Cortex Analyst scored 56.4 / 77.1 on Spider 1.0 under exact match / got-the-facts.
- On Spider 2.0, Cortex Analyst had 27 refusals out of 135 answers.
Key questions
Q: Why do text-to-SQL benchmarks produce different winners under different rules?
A: Because exact match and fact containment reward different things.
Q: What are the signs that a text-to-SQL evaluation is too dependent on metadata quality?
A: High refusal rates, sensitivity to missing join relationships, and large swings between corpora usually indicate that the setup is shaping the score.
Q: How should teams evaluate text-to-SQL tools without relying on misleading leaderboards?
A: Start by matching the benchmark to your environment, then choose the scoring rule that reflects how your users will judge a correct answer.
Practitioner guidance
- Fix the grading rule before you compare vendors Define whether your use case needs exact-match equivalence, fact containment or another metric, then apply the same rule to every candidate.
- Test on your own warehouse and question set Run a small set of real analyst questions against both systems, using your own schema, joins and semantic annotations so the result reflects your environment.
- Track refusals as a separate KPI Record how often each system refuses, why it refuses and whether the refusal is caused by missing metadata, scope limits or an actual inability to answer.
Bottom line: The main risk is not only model quality but benchmark instability, because the same system can look materially better or worse when the scoring rule changes.
Glossary terms: Text-to-SQL, Exact Match, Got-the-Facts, Semantic View
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
This topic was modified 7 days ago by NHI Mgmt Group
27/10/2026 7:17 pm
Benchmark governance is now part of model governance: a text-to-SQL score without its grading rule, corpus and refusal count is not a decision-grade control. The article shows that the same system can move from a 7.2-point lead to a 0.4-point lead simply by changing how correctness is defined. For practitioners, this is the AI equivalent of access review evidence that cannot be reproduced: the measurement itself becomes the risk.
A question worth separating out:
Q: How should security teams govern data access for AI workloads?
A: They should govern AI data access by business purpose, dataset classification, and downstream reuse, not by repository alone. If AI systems can transform or redistribute data, then the entitlement review must cover how the data will be used after access is granted. That requires tighter alignment between IAM, data governance, and AI owners.
👉 Read our full editorial: Text-to-SQL accuracy shifts with the grading rule attached
This post was modified 7 days ago by NHI Mgmt Group