Prompt injection in a public chatbot can expose system prompts fast

At a glance

What this is: Archestra’s case study shows a public-facing chatbot can be compromised through prompt injection in seconds, leading to instruction override, policy abuse, and system prompt leakage.

Why it matters: For IAM, NHI, and AI security teams, the lesson is that exposed conversational interfaces need gateway controls, output filtering, and privilege separation before they are treated as production systems.

Context

Prompt injection is an application-layer attack on large language model interactions, not a flaw in the model weights themselves. The risk appears when untrusted user text is allowed to shape the model’s instructions, priorities, or outputs without a control layer between the user and the privileged prompt.

In this case, the attack surface was a public chatbot exposed on a homepage with no retrieval layer, no tool calls, and no structured output constraints. That is a direct identity and governance issue as well as an AI security issue, because the system prompt and runtime authority became reachable through ordinary user input.

Key questions

Q: What breaks when a public chatbot has no prompt-injection gateway controls?

A: The model can be steered by untrusted user text, which means policy, tone, and even hidden instructions become attacker-controlled variables. Without a gateway that classifies input and constrains output, the assistant can violate policy, leak its system prompt, and amplify small prompts into public incidents.

Q: Why does prompt injection create risk even when credentials are valid?

A: Prompt injection works because the agent uses its own authorised access. The attacker does not need to steal a key if they can steer the agent into using that key for an unintended action. Valid credentials confirm identity, but they do not prove the intent behind each runtime decision.

Q: How should teams separate user input from privileged model instructions?

A: Use a mediation layer that transforms raw user text into a structured request before the privileged model sees it. That boundary lets you validate intent, strip override language, and prevent the model from treating adversarial input as part of its governing instructions.

Q: What should security teams do when a chatbot leaks its system prompt?

A: Treat the leak as a security incident, not a harmless content glitch. Rotate or rewrite the prompt, review logs for repeated probing, assess whether internal names or deployment details were exposed, and add deterministic output checks so the same prompt cannot be replayed back to the public.

Technical breakdown

Why instruction override works in a naked LLM chat flow

A naked LLM behind a chat UI has no separation between trusted instructions and untrusted user content unless the application builds that boundary explicitly. When a prompt contains both policy text and user text in the same context window, the model can treat adversarial user language as a higher-priority instruction if the surrounding controls are weak. The failure is not that the model “understands” the attack, but that the deployment gives the attacker direct access to the same decision space as the system prompt. This is why prompt injection is fundamentally an application control problem. See the [[L3|Prompt Injection Guide]]. See the Prompt Injection Guide.

Practical implication: keep untrusted user text out of the privileged prompt path and classify override attempts before the model sees them.

Why prompt leakage is a distinct failure from policy violation

Policy violation and prompt exfiltration are related but different outcomes. In one case, the model follows malicious instructions and generates disallowed content. In the other, the model reveals the hidden instructions that govern its behaviour, which can expose internal names, operational details, and guardrail logic. Verbatim leakage usually happens when the application passes the system prompt into a response channel without a deterministic filter or a constrained output schema. Once the prompt is disclosed, attackers can tune future prompts against the exact guardrails in use, which makes subsequent abuse easier and more reliable.

Practical implication: scan outputs for prompt fragments before release and treat verbatim system prompt disclosure as a reportable incident.

Why dual-LLM separation changes the trust boundary

Dual-LLM separation places untrusted user input in a quarantined processing path and keeps the privileged model isolated from raw adversarial text. The privileged model receives a structured summary rather than the original prompt, so the attacker cannot directly target the hidden instructions. This pattern matters because it changes the attack from direct instruction manipulation to a mediated request that can be validated, transformed, or rejected first. It is not a silver bullet, but it forces the application to make an explicit trust decision before the privileged context is exposed. In identity terms, it is a form of privilege separation for model interaction. Read AI Agents vs Agentic AI.

Practical implication: put a mediation layer between public input and privileged model context so the assistant never sees raw untrusted instructions directly.

Threat narrative

Attacker objective: The attacker’s objective was to bypass the bot’s controls, force policy-breaking output, and extract hidden prompt instructions for future abuse.

1. Entry occurred when a reporter submitted adversarial text to a public chatbot that accepted arbitrary user input through a homepage chat interface.
2. Credential or control abuse followed when the model accepted an instruction override and treated user text as a new governing rule set.
3. Impact arrived when the bot generated prohibited content and revealed its full system prompt, exposing internal guardrails and deployment details.

Breaches seen in the wild

• EchoLeak (Microsoft 365 Copilot) 2025: A crafted email could make Microsoft 365 Copilot leak data from its context with no click, a zero-click prompt injection fixed as CVE-2025-32711.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Prompt injection is a gateway failure, not a model failure. The article’s incident shows that the real control gap sits between the public interface and the privileged prompt, where untrusted text was allowed to steer a production assistant. That means teams should stop treating the model as the only security boundary and start governing the entry path, output path, and prompt exposure boundary as one control plane.

System prompt leakage creates reusable attacker intelligence. Once the hidden instructions are exposed, the attacker no longer has to guess how the assistant is constrained. That turns a single abuse event into a repeatable tuning exercise, which is why prompt disclosure deserves the same seriousness as credential exposure in an application context. Prompt Injection Guide remains the right reference point because the exploit class is structural, not accidental.

Identity and privilege separation need to extend into agentic interfaces. Even a simple chatbot becomes a privileged runtime when it can speak for an organisation, reveal internal policy, or shape public messaging. That makes the boundary between user input and system authority an identity problem as much as a model safety problem, especially when the interface is exposed on a public domain.

Over-trusting “simple chat” deployments is now a predictable governance error. The article describes a common pattern: a thin wrapper, a long system prompt, and no mediation layer. That is exactly the shape that turns natural language into an attack surface, so the governance mistake is assuming that low complexity means low risk. Practitioners should treat any public LLM endpoint as production access control, not as a prototype interface.

From our research library:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap, according to the State of Secrets in AppSec.
• Read next: Prompt Injection Guide

What this signals

Prompt injection will keep finding public interfaces that expose raw model context. The governance problem is not limited to chatbots that answer questions. Any interface that lets untrusted text shape privileged model behaviour needs a control boundary, otherwise the model becomes the delivery mechanism for the attacker’s instructions.

Model security and identity governance are converging at the prompt boundary. When a public assistant can reveal internal policy, it is operating like a privileged service account with no effective separation between caller and authority. That is why organisations need to design for mediated access, explicit trust transitions, and auditable output, not just safer model prompts.

Detection should focus on override language, prompt replay, and leakage patterns. Teams should watch for repeated instruction-override attempts, sudden shifts in model persona, and verbatim output that mirrors hidden instructions. Those signals often appear before broader abuse becomes visible, which makes them useful for early containment.

For practitioners

• Implement input classification before model execution Screen user text for instruction-override patterns, policy bypass language, and prompt-exfiltration probes before it reaches the privileged prompt.
• Separate untrusted and privileged model paths Route raw user content into a quarantined model or mediation layer, then send only a structured summary to the model that can act on trusted instructions.
• Add deterministic output filtering for prompt leakage Scan every response for fragments of the system prompt, internal names, and guardrail text before the content is released to the public.
• Constrain public chat endpoints to a fixed response schema Use response templates and allowlisted output formats so the model cannot freely emit internal instructions, long-form policy text, or unsupervised prose.

Key takeaways

• The incident shows that a public chatbot can be manipulated through ordinary text input if no gateway controls separate user content from privileged instructions.
• The main operational failure was exposure of hidden prompt material, which turns a single abuse into reusable attacker intelligence.
• Mediation, output filtering, and constrained response schemas are the controls that matter most when a chatbot is exposed to the public internet.

Related NHI Mgmt Group resources: Prompt Injection Guide · AI Agents vs Agentic AI · AI Agent Observability, Audit and Incident Response Guide · EchoLeak (Microsoft 365 Copilot) 2025

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• System Prompt: The system prompt is the core instruction set that defines an LLM’s behaviour, boundaries, and response style during a session. When attackers influence or override it, they are not merely changing text. They are trying to change the model’s governing authority.
• Output Filtering: Output filtering is the control layer that scans AI responses before they are released to the user. It looks for secrets, PII, internal identifiers, and policy violations, and it is essential when models can generate fluent but unsafe disclosures from privileged context.
• Mediated Model Access: Mediated model access is an architecture in which raw user input is transformed, validated, or summarised before reaching a privileged model context. This creates a control boundary between untrusted prompts and trusted instructions, reducing the chance that adversarial text can directly steer model behaviour.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore nhimg.org for resources that connect identity governance to the broader security disciplines your programme depends on.