What is the difference between execution accuracy and logical completeness in text-to-SQL evaluation?

Execution accuracy checks whether the generated query returns the same results as the reference query. Logical completeness checks whether every required condition, join, filter, and aggregation is present in the SQL. A query can score well on execution for one dataset and still omit an important business rule, so both dimensions are needed.

Why Execution Accuracy and Logical Completeness Measure Different Things

Execution accuracy tells you whether the SQL output matches the reference result on the test database. Logical completeness asks a different question: did the query express all of the intended conditions, joins, filters, and aggregations that the task required? That distinction matters because a query can appear correct on one schema instance while still missing a business rule, joining the wrong table, or relying on coincidental data properties rather than the intended logic. For text-to-SQL work, treating those metrics as interchangeable can hide weak generalisation and overstate model quality. In practice, many teams discover the gap only after a query performs well in a benchmark but fails when a schema, value distribution, or reporting rule changes.

How They Diverge in Real Evaluations

Execution accuracy is an outcome metric: if the generated SQL produces the same result set as the gold query on the evaluation database, it passes. Logical completeness is a structure-and-intent metric: it checks whether the query contains everything needed to satisfy the natural-language request, even if the current dataset makes an omission harder to detect. That is why a query can be execution-correct yet logically incomplete. For example, a missing filter may not change the result if no rows violate the condition in the test database, or a mistaken join may still return the right rows because the data happens to be clean. The reverse is also possible in edge cases, where a logically sound query fails execution because of harmless syntactic variation, aliasing, or equivalent formulation differences.

Execution accuracy is sensitive to the specific database state used in testing.
Logical completeness is sensitive to whether every task requirement is represented in the SQL.
One metric can mask errors the other is designed to expose.
Together they help separate accidental correctness from reliable query generation.

That is why evaluation design should not reward only result matching; it should also inspect whether the model captured the full intent of the question. For a broader control perspective on evaluation discipline and traceable criteria, the NIST SP 800-53 Rev 5 Security and Privacy Controls guidance offers useful structure for thinking about consistent measurement and review. The guidance breaks down when the benchmark itself is underspecified or when multiple SQL formulations are equally valid but the evaluator assumes only one shape is acceptable.

Where Benchmarks, Data, and Human Review Change the Interpretation

Tighter evaluation often increases review overhead, requiring teams to balance measurement speed against the risk of missing latent logical errors. This is especially true when the test database is small, skewed, or too clean to reveal omitted predicates. In those cases, execution accuracy can overestimate capability because the database does not exercise the failure mode. Logical completeness is also not a pure string-match problem, because semantically equivalent SQL can be expressed in different ways; a strict structural check may undercount valid answers if it ignores equivalent expressions. The industry does not fully agree on a single best proxy for completeness, so many researchers combine automated scoring with manual inspection on a sample of outputs.

Teams also need to distinguish benchmark success from deployment readiness. A model that passes execution accuracy may still generate queries that are brittle under changing data, incomplete under more realistic business rules, or unsafe for downstream reporting. For that reason, logical completeness is most valuable when the task has hidden constraints, multi-table joins, or aggregation logic where omitted clauses materially change meaning. If the evaluation setup cannot surface those conditions, the metric will be less informative and the result should be treated cautiously.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

NIST CSF 2.0, CIS Controls v8, NIST AI RMF and NIST IR 8596 set the technical controls, while ISO/IEC 42001:2023 define the regulatory obligations.

Framework	Control / Reference	Relevance
NIST CSF 2.0	GV.OV — Oversight	Evaluation needs consistent oversight of quality metrics and review criteria.
Recommendation — Define review criteria for SQL evaluation metrics and verify they reflect both outcome and intent.
CIS Controls v8	16 — Application Software Security	Text-to-SQL evaluation is a software quality control problem tied to secure logic correctness.
Recommendation — Test generated SQL against cases that expose omitted conditions and logic gaps.
NIST AI RMF	ME — Measure	Model evaluation hinges on measurement design that distinguishes accuracy from completeness.
Recommendation — Measure model outputs with complementary metrics that capture both correctness and task coverage.
ISO/IEC 42001:2023	9 — Performance evaluation	AI evaluation governance needs defined performance measures and review of model outputs.
Recommendation — Establish evaluation criteria that separately score SQL result correctness and logical coverage.
NIST IR 8596	A — AI Incident Reporting and Response	Incomplete SQL can create downstream analytic errors that need escalation and correction.
Recommendation — Escalate systematic logical omissions as evaluation defects and track them through remediation.

Practitioner Guidance

What to verify: Check whether the benchmark database can actually expose missing joins, filters, and grouping logic before trusting execution accuracy as a quality signal. If the data is too forgiving, add adversarial or counterfactual test cases that would fail an incomplete query.

Decision rule: Use execution accuracy to measure final answer correctness, but use logical completeness to judge whether the model understood the task. When those two disagree, treat the query as risky rather than successful.

Common mistake: Treating a single high execution score as proof that the model is robust. That usually means the evaluator measured outcomes on one dataset shape, not the completeness of the reasoning that produced them.

Practitioner takeaway: The most reliable text-to-SQL evaluations test both result equivalence and intent coverage, because one can be right while the other is quietly wrong.