Text-to-SQL on your own database: what makes the score real?

TL;DR: A text-to-SQL score only means something when teams write the questions themselves, fix the grading rule before seeing results, run the full set at least three times, and separate refusals from errors, according to Agentic Fabriq. The measurement problem is governance, not model quality.

Editorial analysis by NHI Mgmt Group, based on content published by Agentic Fabriq: “How to Evaluate Text-to-SQL on Your Own Database”.

Key questions

Q: How should enterprises evaluate text-to-SQL systems before putting them into production?

A: Enterprises should evaluate text-to-SQL systems across multiple dimensions, not just accuracy.

Q: Why do text-to-SQL scores change when the rubric changes?

A: Because the same SQL result can be judged differently under exact match, fact-based matching, or human review.

Q: What should teams do when a text-to-SQL system refuses too often?

A: Treat refusal rate as a separate control signal, not as a hidden error count.

Practitioner guidance

• Define the evaluation question set from real workload data Extract questions from BI query logs, backlog items, and support tickets, then tag them by difficulty and refusal class so the sample reflects actual warehouse use.
• Lock the grading rule before any model output Choose exact match, got-the-facts, human judgment, or another explicit rubric before running the test, and keep that rule fixed for every comparison.
• Run the full set at least three times Repeat the same configuration across multiple runs, then compare per-question verdicts to surface instability instead of hiding it inside a single aggregate.

Bottom line: The main risk is not inaccurate SQL alone, but an evaluation process that cannot distinguish a real capability from a well-structured demo.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →