What breaks when an AI chatbot can treat untrusted text as an instruction?

Why This Matters for Security Teams

When a chatbot can treat untrusted text as an instruction, the security boundary shifts from the model itself to everything that feeds it, stores its output, or executes its tool calls. That is why prompt injection is not just a content safety issue; it becomes a workflow integrity issue. The attack path often starts in a harmless-looking message, document, or page and ends with redirected actions, data exposure, or unauthorized automation.

Security teams often miss the risk because the system still “answers correctly” while quietly taking the wrong action. This is especially dangerous in support bots, copilots, and agentic workflows that can open tickets, fetch records, send messages, or modify accounts. The lesson from incidents like the OmniGPT Breach — 34M Conversations Exposed is that once untrusted text can steer downstream behavior, the chatbot becomes a control surface rather than a passive interface. Current guidance from the NIST Cybersecurity Framework 2.0 still applies, but it must be interpreted through the lens of input trust, output handling, and execution authority. In practice, many security teams encounter this only after a chatbot has already relayed the wrong instruction into a live workflow.

How It Works in Practice

The break happens when the system fails to separate data from directives. A model may read pasted text, retrieved documents, emails, tickets, or web content and then produce an output that downstream code treats as actionable. If that output is later parsed as JSON, Markdown, HTML, a command, or a tool request, the attacker has effectively converted content into instruction. The model does not need to be “hacked” in the traditional sense; it only needs to be tricked into obeying the wrong source of authority.

In practice, teams reduce this risk by hardening three layers:

• Input handling: label all external text as untrusted and never let it override system or policy instructions.

• Output handling: treat model output as text, not executable structure, unless it is validated against a strict schema.

• Tool isolation: require explicit authorization before the model can call APIs, send messages, or change records.

That maps closely to how NHI security treats credentials and execution rights. If a chatbot can invoke tools, it needs workload-scoped identity, short-lived authorization, and a clear policy layer. The pattern is consistent with the failures documented in McDonald's McHire AI Chatbot Default Credentials and the broader exposure problem discussed in DeepSeek breach, where access paths and trust assumptions were weaker than the surrounding automation suggested. The practical control point is not the prompt alone but whether untrusted text can influence privileged state changes. These controls tend to break down when retrieval, tool use, and message routing are chained together in one autonomous workflow because the system can no longer reliably tell provenance from instruction.

Common Variations and Edge Cases

Tighter prompt and output controls often increase operational overhead, requiring organisations to balance safety against developer speed and conversational flexibility. That tradeoff becomes especially sharp in customer support bots, internal knowledge assistants, and multi-agent systems where the model must summarize third-party text without acting on it.

Current guidance suggests a few common variations, but there is no universal standard for this yet:

• If the chatbot only drafts text for human review, the main risk is accidental copy-and-paste of hidden instructions into another system.

• If the chatbot can access tools, the risk becomes unauthorized execution and lateral movement through APIs, tickets, or identity workflows.

• If the chatbot handles retrieved content, attackers can poison the retrieval layer so the model sees malicious text as trusted context.

Practitioners should also watch for “format smuggling,” where attacker text is embedded in HTML, tables, code blocks, or metadata that the model later normalizes into action. This is why governance should align with the NIST Cybersecurity Framework 2.0 while also treating model output as an untrusted interface. For AI-facing control design, the OmniGPT breach is a reminder that one unsafe trust boundary can cascade across many sessions and users. Best practice is evolving, but the core rule is stable: if the bot can act, every piece of text it consumes must be assumed adversarial until proven otherwise.

Related resources from NHI Mgmt Group

• What breaks when a model can be persuaded to treat untrusted text as system-level instruction?
• What breaks when an AI coding agent trusts external error reports too much?
• What breaks when telemetry from AI agents includes identity data by default?
• What breaks when AI assistants retain summaries of confidential content?