Text-to-SQL accuracy shifts with the grading rule attached

At a glance

What this is: This first-party benchmark compares Snowflake Cortex Analyst and Databricks Genie across BIRD, Spider 1.0 and Spider 2.0, and shows that the ranking gap changes materially when the grading rule changes.

Why it matters: For IAM and security teams evaluating AI systems, the lesson is that measurement design can distort procurement decisions, so the control question is not only capability but how that capability is being scored, loaded and verified.

By the numbers:

• Databricks Genie scored 37.4 / 53.1 on BIRD mini-dev under exact match / got-the-facts.
• Snowflake Cortex Analyst scored 56.4 / 77.1 on Spider 1.0 under exact match / got-the-facts.
• On Spider 2.0, Cortex Analyst had 27 refusals out of 135 answers.

Context

Text-to-SQL systems do not answer from schema alone. They need business context, join metadata and a grading rule that matches the use case, otherwise a score can describe the harness as much as the product.

This comparison matters because the same answers can produce different outcomes when the evaluation changes, and that is a governance problem as much as a model-quality problem. In identity-adjacent analytics programmes, the decision to trust an AI system depends on how reproducible its results are, not only on whether it looks accurate in a single benchmark.

Key questions

Q: How should teams evaluate text-to-SQL tools without relying on misleading leaderboards?

A: Start by matching the benchmark to your environment, then choose the scoring rule that reflects how your users will judge a correct answer. A leaderboard is only useful as a screening signal. Final selection should use your own schema, your own query patterns, and a grader that reflects business correctness rather than benchmark convenience.

Q: Why do text-to-SQL benchmarks produce different winners under different rules?

A: Because exact match and fact containment reward different things. One system can return the right answer but in the wrong shape, which fails strict equality while still satisfying a looser fact-based test. That means the winner depends on whether the buyer values exact result structure, usable facts, or both.

Q: What are the signs that a text-to-SQL evaluation is too dependent on metadata quality?

A: High refusal rates, sensitivity to missing join relationships, and large swings between corpora usually indicate that the setup is shaping the score. If a system improves dramatically after keys, relationships or semantic hints are added, the benchmark is partly measuring preparation quality rather than model quality.

Q: How should security teams govern data access for AI workloads?

A: They should govern AI data access by business purpose, dataset classification, and downstream reuse, not by repository alone. If AI systems can transform or redistribute data, then the entitlement review must cover how the data will be used after access is granted. That requires tighter alignment between IAM, data governance, and AI owners.

Technical breakdown

Why exact match and got-the-facts diverge in text-to-SQL

Exact match checks whether the generated SQL result has the same rows and columns as the reference answer. Got-the-facts is looser: it accepts an answer that contains the reference result even if extra columns or shape differences are present. That distinction matters because text-to-SQL systems can recover the right fact while still producing the wrong structure for the benchmark. In practice, a product can appear meaningfully better or worse depending on whether the evaluator cares about result shape, factual inclusion, or both.

Practical implication: Choose the grading rule before comparing products, then keep that rule fixed across all candidates.

How semantic context changes text-to-SQL accuracy

These systems are not just schema-to-query translators. They rely on extra business context, such as semantic views in Snowflake or catalog annotations, join relationships and instructions in Databricks. The article shows that missing foreign keys, table limits and semantic modelling choices affected the outcomes, especially on harder schemas. That means the benchmark is partly testing how well the data model was prepared, not only how strong the model is at SQL generation.

Practical implication: Treat semantic modelling and metadata readiness as part of the evaluation, not as invisible setup work.

Why refusal rates are an evaluation signal, not a footnote

A refusal is not a neutral outcome. In this benchmark, refusals on Spider 2.0 were driven mainly by missing join metadata, which means the system could not complete the task under the supplied context. High refusal rates can suppress apparent accuracy and mask whether the underlying problem is model weakness, metadata gaps, or an overly constrained harness. For practitioners, refusals are especially important because they reveal where operational use will stall before a query is ever returned.

Practical implication: Track refusals separately from accuracy so you can distinguish capability gaps from data-preparation gaps.

NHI Mgmt Group analysis

Benchmark governance is now part of model governance: a text-to-SQL score without its grading rule, corpus and refusal count is not a decision-grade control. The article shows that the same system can move from a 7.2-point lead to a 0.4-point lead simply by changing how correctness is defined. For practitioners, this is the AI equivalent of access review evidence that cannot be reproduced: the measurement itself becomes the risk.

Metadata quality is the hidden control surface behind text-to-SQL performance: the strongest signal in this comparison is not which product topped the table, but how much the outcome depended on join metadata, table limits and semantic modelling. That is a governance lesson for any programme that expects AI to operate over enterprise data. If the data model is incomplete, the evaluation will reward the harness, not the system.

The named concept is evaluation drift: a benchmark can tell two defensible stories about the same run when the rule, corpus or loading assumptions change. That makes procurement conversations brittle unless teams separate model capability from evaluation design. For identity and security teams, the practical conclusion is to test systems in the environment they will actually govern, not in a borrowed benchmark frame.

Identity programmes should care because trust decisions increasingly depend on machine-generated answers: when AI is used to interrogate operational data, the question is not only whether the output is plausible, but whether the evaluation pipeline is stable enough to justify access, approval or remediation decisions. IAM and identity provider controls matter here because the organisation still has to decide who can rely on the system, who can modify its context, and who can accept its answers.

Vendor benchmarks remain useful, but only as one input into an evidence chain: the article’s disclosure around its own system, its own harness and its own load issues is the right instinct, because benchmark credibility depends on traceability. The practitioner takeaway is simple: measure the product, the metadata and the test design together, then treat any single headline score as provisional.

What this signals

Evaluation drift is the hidden procurement risk: when a benchmark’s rule changes the winner, the organisation is not just buying model performance, it is buying an evaluation philosophy. That should push buyers to ask whether their own decision criteria are stable enough to survive contact with production data and messy schemas.

The stronger programme pattern is to evaluate AI data assistants as governed workflows, not standalone models. That means measuring refusals, metadata completeness and answer reproducibility alongside raw accuracy, because those are the controls that determine whether the system can be trusted in operational use.

For practitioners

• Fix the grading rule before you compare vendors Define whether your use case needs exact-match equivalence, fact containment or another metric, then apply the same rule to every candidate.
• Test on your own warehouse and question set Run a small set of real analyst questions against both systems, using your own schema, joins and semantic annotations so the result reflects your environment.
• Track refusals as a separate KPI Record how often each system refuses, why it refuses and whether the refusal is caused by missing metadata, scope limits or an actual inability to answer.

Key takeaways

• The main risk is not only model quality but benchmark instability, because the same system can look materially better or worse when the scoring rule changes.
• The article shows that refusal rates and metadata gaps can distort apparent performance, especially on more complex schemas.
• Practitioners should validate text-to-SQL tools on their own data, with fixed rules and reproducible setup, before relying on headline accuracy claims.

Related NHI Mgmt Group resources: NHI Security Platform Buyer's Guide · Secrets Management Buyer's Guide · IAM and Identity Provider Buyer's Guide

Key terms

• Text-to-SQL: An approach that converts natural language questions into SQL queries against structured databases. It lets non-technical users ask business questions in plain language while the system generates database commands. Security teams should govern it carefully because it can expose sensitive records if permissions and query boundaries are too broad.
• Exact Match: Exact match is a scoring rule that only counts an answer as correct when its rows and columns match the reference result. It is strict and useful for precision testing, but it can understate value when a query returns the right facts in a different structure.
• Got-the-Facts: Got-the-facts is a looser scoring rule that accepts an answer when it contains the reference result, even if extra columns or formatting differences are present. It is useful when factual retrieval matters more than shape, but it can overstate practical query quality.
• Semantic View: A semantic view is a governed abstraction that presents data in business-friendly terms for analytics or AI use. It sits between raw source tables and higher-level applications, helping teams standardize meaning while still needing traceability back to the underlying data assets and control framework.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security and the control thinking that underpins trustworthy access decisions. It is suited to practitioners who need to connect identity governance with broader security decisions and evidence-based evaluation.