Treat refusal rate as a separate control signal, not as a hidden error count. If the system declines questions that it could answer, coverage is too low. If it answers too many risky questions, the brake is ineffective. Compare strict and permissive runs to see whether the refusals are actually protecting quality.
Why refusal rate is a quality control signal, not just a nuisance metric
A text-to-SQL system can refuse for two very different reasons: it is protecting you from unsafe or low-confidence output, or it is declining useful requests that it should handle. Teams should treat refusals as a measurable behaviour with its own threshold, because both under-refusal and over-refusal can degrade trust in the system.
The key question is not whether the model says “no” often, but whether the refusals line up with the kinds of queries you actually want blocked. If refusal is high on ordinary, answerable questions, coverage is too narrow. If refusal is low on ambiguous or high-risk queries, the guardrail is too weak.
How to tell whether the system is refusing the right questions
Start by separating request classes into at least two buckets: questions the system should answer, and questions it should decline or route elsewhere. Then compare refusal behaviour across both buckets. A healthy system will be selective, not uniformly cautious, and the balance should reflect the domain, the schema, and the downstream harm of a bad SQL result.
Compare strict and permissive runs on the same prompt set. If stricter settings barely reduce errors but sharply increase unnecessary refusals, the control is too aggressive. If permissive settings produce more answers but also more obviously unsafe or wrong SQL, the refusal gate is doing useful work and should stay in place, even if it frustrates users.
What teams should change when refusals are too frequent
Frequent refusals usually point to one of three issues: the prompt policy is too conservative, the system lacks enough schema or business context to answer confidently, or the evaluation set is too broad for the current capability level. Teams should identify which of those is driving the behaviour before changing thresholds, because lowering refusal sensitivity without improving grounding can simply convert refusals into low-quality answers.
For production systems, the right fix is often better retrieval, clearer SQL generation constraints, or narrower query routing rather than a blanket policy change. If the system repeatedly refuses on benign prompts that are within scope, it may need better examples, stronger schema linking, or a more explicit definition of what counts as unsupported versus unsafe.
Risk and Threat Considerations
Over-refusal can become an operational risk when users stop trusting the system and work around it with manual queries or shadow tools. Under-refusal is the opposite failure mode: the system will answer queries it should have blocked, which can expose sensitive data, create misleading results, or produce SQL that hits the wrong tables and gives a false sense of confidence.
Failure mechanism: A blunt refusal policy collapses distinct cases, such as out-of-scope requests, low-confidence requests, and genuinely dangerous requests, into the same outcome. That makes the control look safe while silently reducing utility, or it makes the system look helpful while allowing risky queries through.
Impact: Teams lose either coverage or safety, and in both cases the refusal signal stops being a reliable indicator of system quality. Once that happens, operators cannot tell whether the model is protecting users or merely avoiding hard questions.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST SP 800-53 Rev 5 and OWASP ASVS set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev 5 | AU-6 — Audit Review, Analysis, and Reporting | Refusal trends are a quality signal that should be reviewed for control effectiveness. |
| SI-10 — Information Input Validation | Text-to-SQL input handling must reject unsafe or unsupported requests before query generation. | |
| Recommendation — Review refusal telemetry to distinguish protective denials from avoidable coverage gaps. Validate prompts and query intents before generation to block unsafe SQL paths. | ||
| OWASP ASVS | V15 — Secure Coding and Architecture | The answer concerns balancing safe capability and failure handling in a production system. |
| Recommendation — Design the system so safe requests succeed while unsafe requests are consistently declined. | ||
Practitioner Guidance
What to verify: Check refusal rate separately for normal business questions, ambiguous questions, and questions that touch sensitive joins, restricted tables, or destructive operations. The refusal pattern should differ across those groups, otherwise the control is probably too coarse.
Decision rule: If refusals rise but answer quality does not improve, tune the prompt, routing, or grounding before lowering the brake. If answers expand but error or leakage risk rises, keep the brake and improve coverage through better context and retrieval.
Practitioner takeaway: Treat refusal as a calibrated safety boundary, not a success metric by itself. The goal is selective refusal, enough to block harmful or unsupported SQL without suppressing legitimate work.
Related resources from NHI Mgmt Group
- What should teams do if a certified biometric system still creates too many false rejects?
- How should security teams validate web applications when attackers often start there too?
- Why do identity governance programmes often fail when teams keep too much in house?
- What are the signs that a text-to-SQL evaluation approach is too weak for production?