TL;DR: The score can shift with the grading rule, the question set, the repetition count, and how refusals are counted, according to Agentic Fabriq. Evaluation design has to be treated as a governance control so stability and deferrals are reported separately.
Editorial analysis by NHI Mgmt Group, based on content published by Agentic Fabriq: “How to Evaluate Text-to-SQL on Your Own Database”.
Key questions
Q: Why do text-to-SQL scores change when the rubric changes?
A: Because the same SQL result can be judged differently under exact match, fact-based matching, or human review.
Q: What should teams do when a text-to-SQL system refuses too often?
A: Treat refusal rate as a separate control signal, not as a hidden error count.
Q: How should enterprises evaluate text-to-SQL systems before putting them into production?
A: Enterprises should evaluate text-to-SQL systems across multiple dimensions, not just accuracy.
Practitioner guidance
- Define the evaluation question set from real workload data Extract questions from BI query logs, backlog items, and support tickets, then tag them by difficulty and refusal class so the sample reflects actual warehouse use.
- Lock the grading rule before any model output Choose exact match, got-the-facts, human judgment, or another explicit rubric before running the test, and keep that rule fixed for every comparison.
- Run the full set at least three times Repeat the same configuration across multiple runs, then compare per-question verdicts to surface instability instead of hiding it inside a single aggregate.
Bottom line: The main risk is not inaccurate SQL alone, but an evaluation process that cannot distinguish a real capability from a well-structured demo.
Glossary terms: Grading Rule, Refusal Rate, Case Flip Rate, Answerable Question Set
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
Evaluation governance is the real control plane for text-to-SQL. The article is right to treat the scoring rubric, question set, and refusal handling as the primary variables. In practice, the system under test is only as trustworthy as the measurement process that surrounds it. For identity and data teams, that means the evaluation itself needs ownership, versioning, and repeatability before anyone treats the result as operational evidence.
A question worth separating out:
Q: How many questions are enough for a useful text-to-SQL evaluation?
A: A few dozen carefully chosen questions can diagnose what to fix, but they cannot prove one configuration is better than another. The useful floor is enough to separate schema, definition, and refusal failures. A larger sample is needed only when you want to compare systems with statistical confidence.
👉 Read our full editorial: Text-to-SQL evaluation is only honest when the rubric comes first