Execution Accuracy

What execution accuracy measures

Execution accuracy is an outcome-based metric for text-to-SQL systems. Instead of comparing a generated query to a reference string, it checks whether both queries return the same result on a database. That makes it better than exact-match scoring for judging semantic correctness, but it still depends on the test data and can miss inefficient or logically brittle SQL.

Because the metric is judged by query results, it is especially useful when surface-form differences do not matter. Two SQL statements can be syntactically different yet semantically equivalent, so execution accuracy captures the practical question: did the model produce the right answer on this database state?

How execution accuracy is evaluated

Evaluation usually means running the generated query and the reference query against the same database instance, then comparing their outputs. If the result sets match under the benchmark's rules, the prediction counts as correct. This is why execution accuracy is often used in academic and benchmark settings for text-to-SQL, where the goal is to measure whether the system can answer the user's question correctly, not merely imitate the reference query.

The details of comparison matter. Some benchmarks treat ordering as irrelevant, others require exact row and column equivalence, and some normalise types or duplicate handling. Those choices can change the score even when the underlying SQL is close in intent, so the metric should always be interpreted alongside the benchmark protocol.

Why execution accuracy is stronger than string matching

String-based metrics reward lexical similarity, which can overvalue copied structure and undervalue a correct rewrite. Execution accuracy avoids that trap by focusing on the returned answer. A query that uses different joins, aliases, or clause ordering may still be fully correct if it produces the same result as the reference.

That said, the metric measures correctness only for the tested database instance. A query may pass because the sample data happens to hide a logic flaw, an overly broad filter, or an accidental equivalence. In other words, execution accuracy is a better correctness signal than exact match, but it is not proof that the SQL generalises or is optimal.

What execution accuracy does not tell you

Execution accuracy does not capture performance, maintainability, or semantic robustness beyond the observed output. A query may be correct but slow, unstable across different data distributions, or dependent on accidental properties of the benchmark database. It can also fail to reveal logic errors that only appear on another dataset or after schema or content changes.

For that reason, high execution accuracy should be read as evidence of functional success, not as a complete quality score. In production settings, engineers still need to assess cost, latency, query plan quality, and whether the logic remains correct when the underlying data evolves.

Risk and Threat Considerations

Execution accuracy can be misleading when the evaluation database is too small, too clean, or too convenient. A model may appear reliable while actually depending on dataset-specific coincidences, which creates a false sense of confidence in downstream SQL generation systems.

Failure mechanism: The generated query returns the right answer on the benchmark data even though its joins, predicates, or aggregations are fragile and would fail on different data shapes or volumes.

Impact: Teams may ship a text-to-SQL system that scores well in testing but produces incorrect analytics, hidden data-quality errors, or unstable behaviour in real use.

Practitioner Guidance

What to watch for: Treat execution accuracy as one evaluation layer, not the only one. Pair it with schema-aware review, robustness checks, and targeted cases that probe edge conditions such as duplicates, null handling, joins, and value ambiguity.

Practitioner takeaway: The best use of execution accuracy is as a semantic correctness signal, with broader validation needed before you trust a generated SQL system operationally.

Related resources from NHI Mgmt Group

• Execution (MITRE ATT&CK Tactic)
• What is the difference between execution accuracy and logical completeness in text-to-SQL evaluation?
• What is the difference between SoD accuracy and audit defensibility?
• When should organisations sandbox code execution in agentic platforms?