Common signs include completing diagnostic templates, reflecting user-supplied text in authoritative messages, revealing internal configuration details, and taking action on content that has not been validated or separated from system instructions. If the workflow cannot clearly distinguish user content from control data, the assistant is already mixing trust boundaries.
How to tell when instruction and data have been mixed
One of the clearest warning signs is that the assistant starts behaving as if user content were a control signal instead of input to be evaluated. That often shows up when it follows embedded instructions, rewrites policy or task scope from the prompt, or treats quoted text as if it came from a higher-trust source. A sound assistant should preserve separation between content it is analysing and instructions it is meant to obey.
Another sign is refusal to preserve boundaries in the output. If the model completes a template that was supplied by the user, echoes back sensitive instructions as if they were authoritative, or carries user-provided directives into its own operational answer, it is no longer distinguishing data from control.
That failure matters because it is the same class of weakness that lets Enterprise AI Copilot Security Guide and EchoLeak (Microsoft 365 Copilot) 2025 describe: untrusted content can become an instruction path when the workflow does not enforce clear trust boundaries.
What unsafe behavior looks like in practice
A practical test is whether the assistant begins to act on content before validating it. If it reveals internal configuration details, discloses hidden prompt material, or uses user-supplied text to drive downstream decisions without clear sanitisation or separation, that is a strong sign the boundary has failed. The issue is not just what the assistant says, but whether it is treating externally supplied text as an authority source.
The same pattern appears when an assistant reflects the user’s text back in an authoritative tone. That can make fabricated instructions, malformed templates, or injected policy language look validated when they are not. Once the model starts treating content as trusted because it is present in the conversation, any later action can inherit that mistaken trust.
This is why AI Coding Agents Security Guide and Sentry MCP Agentjacking 2026 are useful analogues, because both show how agentic systems can be manipulated when tool output, developer context, or user-supplied content is allowed to steer execution without validation.
What the assistant should do instead
The right behavior is to treat all untrusted input as content to inspect, not instruction to follow. That means the assistant should keep system rules, developer instructions, tool outputs, and user content in distinct handling paths, and it should avoid taking action unless the content has been validated against the intended role of the field or channel. When those paths are blurred, the model may still sound coherent while silently violating trust boundaries.
For workflows that involve agents, connectors, or tool calls, the safer pattern is to require explicit separation at the interface level, not just better wording in the prompt. The assistant should know which parts of the exchange are data, which are control, and which parts are allowed to trigger action. That distinction is especially important when user content can shape configuration, diagnostics, or operational decisions.
Current guidance from OWASP API Security Top 10 and OWASP Cheat Sheet Series reinforces the same principle in adjacent systems: validate input, enforce object and function boundaries, and never let caller-controlled data define privileged behavior.
Risk and Threat Considerations
When an assistant treats untrusted input as trusted instruction, the failure is not cosmetic, it creates a direct path to prompt injection, hidden policy override, data exposure, and unintended tool use. In agentic workflows, that can turn a malicious string into an execution trigger, especially if the model is allowed to call tools, browse content, or reveal internal state.
Failure mechanism: The assistant does not separate instruction channels from content channels, so attacker-controlled text is interpreted as higher-trust guidance or is propagated into action.
Impact: The assistant may leak internal prompts or configuration, follow attacker-written steps, or take actions that should have required validated, trusted input.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP API Security Top 10 addresses the attack and risk surface, while OWASP ASVS, NIST SP 800-53 Rev 5 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP API Security Top 10 | API8 — Security Misconfiguration | Trust-boundary failures often stem from weak request handling and policy separation. |
| Recommendation — Enforce strict input handling and object boundaries so caller data cannot steer privileged behavior. | ||
| OWASP ASVS | V2 — Validation and Business Logic | The question is about validating untrusted content before it can influence logic or action. |
| Recommendation — Validate user-supplied content before it can affect logic, templates, or downstream actions. | ||
| NIST SP 800-53 Rev 5 | SI-10 — Information Input Validation | This is directly about preventing untrusted input from being treated as trusted instruction. |
| AC-3 — Access Enforcement | The issue becomes harmful when content can cross into privileged actions without enforcement. | |
| Recommendation — Apply input validation controls to separate untrusted content from executable instruction. Enforce privilege boundaries so untrusted content cannot trigger protected actions. | ||
| NIST CSF 2.0 | PR.AA-01 — Identities and credentials are issued, managed, verified, revoked, and audited | Prompt and tool trust failures often involve unsafe use of credentials or delegated access. |
| Recommendation — Manage delegated access so content cannot inherit authority without verification. | ||
Practitioner Guidance
What to verify: Check whether the assistant can explain, in a deterministic way, which inputs are instructions, which are data, and which can trigger tool use. If it cannot demonstrate that separation, treat the workflow as unsafe even when its answers look plausible.
What practitioners underestimate: Many teams look only for obvious jailbreaks, but the more common failure is quiet boundary collapse, where the system appears to answer normally while internal trust decisions have already been compromised.
Practitioner takeaway: The real test is not whether the assistant sounds confident, it is whether it preserves trust boundaries under adversarially written content and refuses to elevate user text into control.
Related resources from NHI Mgmt Group
- What is the difference between using AI as a coding assistant and treating it as a trusted developer?
- What are the signs that an AI agent is being redirected by trusted input instead of acting within its normal job pattern?
- What are the signs that an AI assistant is too deeply integrated into trusted content flows?
- Who is accountable when untrusted project configuration changes what an AI assistant sees?