What breaks when AI coding tools can turn untrusted content into shell commands?

Where the trust boundary collapses between prompt text and execution

When AI coding tools can convert untrusted content into shell commands, the problem is not simply that the tool is “too helpful.” The deeper issue is that text used for suggestion or summarisation starts behaving like instruction, which breaks the assumption that the operator is the only party authorising execution. That is a governance and control problem as much as a technical one, because approval prompts, review steps, and allow-lists are only meaningful when they validate human intent rather than parsed content. For identity and access teams, this is a familiar pattern of delegated authority going wrong, except the delegation is happening through text handling rather than a classic account or token. In practice, many security teams encounter the failure only after a repository, ticket, or document has already influenced an execution path that was treated as safe.

The issue is especially important where coding assistants can read local files, repository content, issue text, or pasted instructions and then emit commands that run with the user’s privileges. That makes the content source part of the effective trust model, even when no one intended it to be. This is why a simple “approve before run” step is not enough if the tool’s interpretation layer can be steered by untrusted input. For readers evaluating machine identity and workflow controls, the relevant question is not whether the command was typed by a person, but whether the content that shaped it was authorised to influence execution.

OWASP Non-Human Identity Top 10

How AI coding tools turn content into action, and where the model fails

These tools usually sit between a natural-language or repository input layer and an execution layer. A user asks for help, the assistant inspects code, comments, docs, or issue text, and then proposes a command, patch, or automation step. If the assistant is allowed to treat untrusted content as a source of operational intent, the output can inherit the content’s hidden instructions rather than the user’s explicit objective. That creates a classic trust-boundary mismatch: the parser sees text, the execution layer sees authority.

The breakage becomes practical in a few common ways:

• A comment or README fragment is interpreted as a command fragment or a reason to run a command.
• A malicious repository embeds instruction-like text that nudges the assistant toward destructive or exfiltrating actions.
• An approval dialogue is presented after the assistant has already shaped the command, so the user reviews a consequence rather than the original intent.
• Local context such as scripts, package metadata, or prompt injection inside documents changes the command plan without a clear operator decision.

That matters because shell commands have immediate side effects: file modification, credential exposure, environment discovery, package installation, or network calls. Once execution is coupled to untrusted text, the tool is no longer merely assisting with syntax. It becomes an interpreter of intent, and that is where safety assumptions tend to fail. The control question is whether the assistant is constrained to treat untrusted content as data, or whether it can promote that content into action. If it can do the latter, the review step is checking output, not authority, and that distinction is what breaks the model.

Good implementations separate read access from execution authority, constrain what content can influence command construction, and make command generation visible before any side effect occurs. Where tools blend retrieval, summarisation, and execution, the boundary is weakest, especially in workflows that automatically trust repository context or pasted instructions.

When the usual safeguard breaks down in real workflows

Tighter confirmation prompts often improve safety, but they also increase friction, which means organisations must balance usability against the risk that the prompt itself is no longer the true control. When the assistant can be steered by a document, issue, or repo file, a human may still click “approve” while never seeing the hidden instruction that shaped the command. That is a real tradeoff: stronger guardrails can slow development, but weak guardrails can create a false sense of review.

The main edge cases are workflows that appear local and low risk but are actually high trust. Examples include agentic IDE features, repository-wide assistants, automation that reads tickets or comments, and tools that can invoke the shell from within a chat session. The guidance is different when the tool only suggests a snippet versus when it can launch commands, modify files, or access secrets. Guidance versus consensus here is still evolving, but there is broad agreement that any path from untrusted text to execution should be treated as a sensitive trust boundary, not a convenience feature.

Where teams often misjudge the situation is assuming that “we showed the command to the user” is sufficient. That is only true when the command was built from trusted intent and the display is legible enough for meaningful review. If the content source can influence command selection, the approval is validating a potentially compromised plan rather than an independently authored action. The break becomes most severe when the assistant has access to credentials, package managers, or deployment environments, because then text injection can lead to operational changes far beyond the original request.

Practitioner Guidance

What to prioritise: Treat any path from untrusted text to shell execution as a high-risk trust boundary, even when the tool is framed as “developer productivity.” The first control objective is to prevent hidden instructions from shaping command generation in the same context that can execute them.

What to verify: Verify whether the assistant can distinguish data from instruction in every input source it reads, including markdown, comments, issue text, and repository files. If the tool cannot prove that distinction, assume approval prompts are advisory rather than authoritative.

Decision rule: If a tool can both ingest untrusted content and emit commands with user privileges, require a design that makes execution intent explicit and reviewable before any side effect occurs. If it cannot enforce that separation, restrict it to read-only assistance.

Practitioner takeaway: The real failure is not command generation itself, but the loss of authorship integrity at the moment untrusted text is allowed to influence execution.

Related resources from NHI Mgmt Group

• What breaks when AI agents can call tools after reading untrusted content?
• What breaks when an AI assistant can access private data and untrusted content at the same time?
• What breaks when an autonomous assistant can read untrusted content and execute tools in the same session?
• What breaks when an AI coding agent can write files that host tools later trust?