How many questions are enough for a useful text-to-SQL evaluation?

A few dozen carefully chosen questions can diagnose what to fix, but they cannot prove one configuration is better than another. The useful floor is enough to separate schema, definition, and refusal failures. A larger sample is needed only when you want to compare systems with statistical confidence.

What counts as enough questions for a useful text-to-SQL evaluation?

The useful threshold depends on what you want the evaluation to tell you. A few dozen well-selected questions can reveal which failure modes dominate, especially schema interpretation, definition handling, and refusal behaviour. That is enough for diagnosis, but not enough to claim one prompt, model, or configuration is better than another with confidence.

For text-to-SQL, the question set should be built to expose distinct classes of error, not to mimic a benchmark leaderboard. If every question looks similar, the evaluation will miss brittleness in joins, aggregation, filters, temporal logic, and business-rule interpretation. If the set is too small, a system can appear strong simply because it was not tested on the cases that break it.

The right answer is therefore less about raw count than about coverage. You want enough questions to separate whether the problem is schema grounding, query construction, ambiguous wording, or safe refusal. Once those categories are represented, adding more questions mainly improves confidence, stability, and comparability across systems rather than changing the basic diagnosis.

Why a small set can still be useful

A compact evaluation is valuable when the goal is debugging. A carefully curated set of 20 to 50 questions can quickly show whether failures cluster around schema naming, column selection, join paths, nested logic, or natural-language ambiguity. That makes it practical for iterative prompt or pipeline tuning, because each miss points to a concrete engineering fix.

The key is to include variety deliberately. Mix straightforward lookup questions with multi-table joins, calculated metrics, negation, time windows, and edge cases that stress definition choice. If the set covers those dimensions, a small sample can still be operationally useful because it tells you where the system is fragile, even if it cannot estimate overall performance precisely.

When you need a larger sample

You need a larger sample when the question is comparative rather than diagnostic. If you want to know whether one model is better than another, or whether a configuration change produced a real improvement, the evaluation must be large enough to reduce noise from question selection. Otherwise, a few lucky or unlucky questions can swamp the result.

That is especially true when the performance gap is small. In text-to-SQL, two systems may both look acceptable on easy questions but diverge on harder ones. A larger sample helps distinguish a genuine improvement from random variation, and it reduces the risk of optimising for a narrow test set that does not reflect real usage.

When comparison is the goal, the better question is not “how many is enough?” but “how many are needed to make the decision reliable?” That usually means more than a few dozen, and often a stratified set that balances difficulty, schema complexity, and question type rather than simply increasing volume.

Risk and Threat Considerations

Text-to-SQL evaluations can create false confidence when the sample is too small or too repetitive. The main risk is overestimating robustness, then deploying a system that handles familiar patterns but fails on uncommon schemas, ambiguous terms, or business-rule exceptions that matter in production.

Failure mechanism: A narrow test set under-samples the hard cases, so weak behaviour is never observed. The evaluation then rewards surface accuracy on a few examples instead of durable performance across the query patterns the system will actually face.

Impact: Teams may ship a system that looks dependable in review but produces wrong queries, silent analytic errors, or overconfident refusals once it encounters real user questions and unfamiliar database structures.

Practitioner Guidance

What to verify: Make sure the set includes the failure modes you most need to detect, not just the questions that are easiest to write. For text-to-SQL, that usually means at least one pass each over schema grounding, definition ambiguity, multi-step joins, aggregation, and refusal behaviour.

Decision rule: If you are trying to debug the system, stop when the set is large enough to show recurring failure patterns. If you are trying to compare systems, expand the set until the result is stable enough that a few questions moving either way would not change the decision.

Practitioner takeaway: A few dozen questions can be enough to find what is broken, but not enough to prove superiority; treat small evaluations as diagnostic tools, and use larger, balanced samples only when you need a defensible comparison.