What breaks when LLM outputs are used directly in SQL queries or shell commands?

Directly reusing model output in executable contexts can convert a generation error into a security incident. A malicious or malformed response may trigger injection, destructive commands, or unintended data access. Safe handling requires validation, escaping, allowlisting, and separate execution controls before any output reaches a database, terminal, or automation step.

Why This Matters for Security Teams

LLM output is not just text when a system treats it as a SQL fragment or a shell command. At that point, a model hallucination, prompt injection, or maliciously crafted response can become code execution, data destruction, or unauthorized data exposure. This is especially dangerous in agentic workflows where an AI agent can chain tool calls, reuse context, and amplify one bad token into a larger incident. Guidance in the OWASP Agentic AI Top 10 and NIST AI Risk Management Framework both point to the same operational truth: output must be treated as untrusted input until it is validated, constrained, and separated from execution.

NHIMG research on the AI Agents: The New Attack Surface report shows why this matters at enterprise scale: 80% of organisations report their AI agents have already performed actions beyond their intended scope, including revealing access credentials. That is the failure pattern security teams see when a seemingly harmless generated string is passed straight into a database client or a terminal. In practice, many security teams encounter this only after the model has already been trusted to execute something it should never have been allowed to decide.

How It Works in Practice

The core break is boundary collapse. SQL engines and shells are interpreters, not passive storage formats. If an application concatenates model output into a query or command, the model no longer just generates language, it influences execution. A response like a clause, flag, operator, or delimiter can change meaning at runtime. That is why string building is dangerous even when the output looks plausible.

Safer designs split generation from execution. The model should produce structured intent, not executable syntax. A policy layer then decides whether the requested action is allowed, and the application maps that approved intent to a parameterised query or a fixed command template. This is aligned with the direction of the NIST AI 600-1 Generative AI Profile, which emphasises controlled interfaces, and with NHIMG guidance in OWASP NHI Top 10, where execution risk is treated as a governance issue, not just a coding issue.

• Use parameterised SQL and never concatenate raw model output into query strings.
• Use allowlisted command templates, not free-form shell execution.
• Validate type, length, charset, and semantic intent before any execution step.
• Keep the model outside the trust boundary for write, delete, and admin actions.
• Log the original prompt, model output, and the final execution decision separately.

For automation-heavy environments, the shell should be treated as a privileged control plane, not a convenience API. The same applies to database privileges: even if the model is compromised, the runtime should not possess broad destructive rights. These controls tend to break down when teams let the model compose ad hoc SQL for analytical workflows or invoke shell tools inside long-lived agent loops because the application loses deterministic control over what is executed.

Common Variations and Edge Cases

Tighter execution controls often increase development overhead, requiring organisations to balance velocity against the need for deterministic behaviour. That tradeoff becomes sharper in environments where users expect natural-language flexibility, such as data exploration tools, DevOps copilots, and autonomous remediation agents.

There is no universal standard for this yet, but current guidance suggests treating different execution paths differently. Read-only analytics can sometimes tolerate generated SQL if it is parsed, normalised, and constrained to a safe subset. Privileged operations should be much stricter: shell access, DDL changes, file writes, and system administration tasks should usually require fixed workflows, human approval, or a separate policy engine such as those discussed in the CSA MAESTRO agentic AI threat modeling framework.

Edge cases also matter. Shell metacharacters, multi-statement SQL, encoding mismatches, and hidden prompt injection all create ways for the output to behave differently from what the reviewer expects. When the assistant is embedded in a workflow with broader permissions, the risk grows further because the same output can trigger lateral movement or data exfiltration across chained tools. NHIMG coverage of the LLMjacking: How Attackers Hijack AI Using Compromised NHIs pattern shows how quickly attacker use of borrowed identities turns a bad output into a larger compromise.

Related resources from NHI Mgmt Group

• What breaks when LLM output is used directly in application logic without validation?
• What breaks when organisations trust LLM outputs too much?
• What breaks when an LLM can call raw backend APIs directly?
• What breaks when an LLM is used to judge its own safety?