The approval model breaks. If markdown, comments, or embedded instructions can influence execution, then allow-lists and approval prompts no longer validate human intent. The control failure is a trust boundary mismatch between text parsing and command execution, which means a malicious repository or document can trigger actions the user never knowingly authorised.
Why This Matters for Security Teams
AI coding tools change the trust model because they can turn text into execution without a clean human decision point. Once markdown, comments, issue text, or retrieved snippets can influence shell commands, standard approval prompts stop proving intent and start validating whatever the model inferred. That is why the core failure is not simply poor sanitization, but a trust boundary mismatch between untrusted content and command execution.
Security teams should treat this as a governance and runtime control problem, not a UI problem. Guidance from the NIST Cybersecurity Framework 2.0 still applies, but it must be translated into agent-safe execution patterns with stronger separation between data ingestion and action. NHIMG has documented how quickly abuse spreads once AI tools are allowed to act on misleading inputs in cases like the Amazon Q AI Coding Agent Compromised incident and the Replit AI Tool Database Deletion case.
In practice, many security teams discover the risk only after an agent has already transformed a harmless-looking prompt, issue comment, or repository note into a destructive shell action.
How It Works in Practice
The failure usually starts when an AI coding assistant has both read access to untrusted content and write or execute privileges in the same workflow. A poisoned README, dependency note, ticket, or code comment can be surfaced by retrieval, copied into context, and then interpreted as an instruction. If the tool can call a shell, patch files, or run build commands, the model may chain those steps together before a person sees the final action.
Current best practice is evolving toward explicit separation of roles: text ingestion is treated as untrusted input, while command execution is gated by a second control plane. That means deny by default, keep shell access out of the model unless strictly required, and use scoped execution wrappers that validate the exact command, working directory, and target files at runtime. The DeepSeek breach illustrates how quickly sensitive content can be exposed when trust assumptions fail inside AI pipelines.
- Require explicit command serialization, so the model cannot freely concatenate text into shell syntax.
- Use ephemeral credentials and short-lived task tokens instead of long-lived developer secrets on the agent host.
- Separate retrieval, reasoning, and execution into distinct layers with audit logging at each boundary.
- Inspect prompts and tool calls for instruction injection patterns, especially when content comes from repositories or ticketing systems.
- Require human review for destructive actions, but validate the actual command rather than the model’s explanation of it.
These controls tend to break down in high-churn developer environments where agents are allowed to operate across many repositories, build systems, and CI runners because the blast radius expands faster than review gates can keep up.
Common Variations and Edge Cases
Tighter command gating often increases developer friction, requiring organisations to balance speed against the risk of prompt-to-shell abuse. That tradeoff becomes more visible in multi-agent workflows, where one agent writes code, another runs tests, and a third prepares deployment steps. Guidance is not fully settled here, but current consensus favors context-aware authorisation and just-in-time execution over broad persistent access.
Edge cases matter. A tool may be safe in a local editor but unsafe in CI because the same prompt can reach build scripts, package managers, or deployment hooks. Markdown rendering, hidden HTML, and copied terminal snippets can also become injection paths if the assistant treats them as instruction-bearing text. The LLMjacking: How Attackers Hijack AI Using Compromised NHIs research shows why identity and credential abuse compound the problem once agents inherit more authority than intended.
For that reason, the safer pattern is not “trust the model less” but “trust the execution environment less.” The best controls are runtime policy checks, short-lived identity, and strict command allow-listing on the actual tool boundary.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, CSA MAESTRO and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A2 | Prompt and tool injection can convert untrusted text into actions. |
| CSA MAESTRO | TA3 | Agent-to-tool execution needs runtime guardrails and policy checks. |
| NIST AI RMF | GOVERN | Autonomous command generation creates accountability and oversight risk. |
| OWASP Non-Human Identity Top 10 | NHI-04 | Short-lived, scoped identities reduce blast radius for agent execution. |
| NIST CSF 2.0 | PR.AC-3 | Access enforcement must reflect runtime context, not static trust. |
Block instruction injection by separating untrusted content from tool execution and validating each tool call.
Related resources from NHI Mgmt Group
- What breaks when AI agents can call tools after reading untrusted content?
- What breaks when an AI assistant can access private data and untrusted content at the same time?
- What breaks when an autonomous assistant can read untrusted content and execute tools in the same session?
- What breaks when agentic coding tools are allowed to trust repository content by default?
