AI agent prompt injection exposed full system prompts in testing

At a glance

What this is: Escape shows that a reframed prompt injection can bypass an AI agent guardrail and expose the agent's system prompt, tools, rules, and session metadata.

Why it matters: This matters because IAM and AI security teams need controls that govern what an agent can reveal and invoke, not just what text it accepts at the prompt boundary.

Context

Prompt injection is a control-evasion problem in AI systems where the attacker tries to make the model or agent ignore its intended instructions. In this case, the issue was not access to the assistant itself, but disclosure of the system prompt, tool inventory, and conversation metadata after an authenticated request was reframed.

For practitioners, the key governance gap is that many agent defenses still assume the first blocked request is the end of the test. Once an assistant can be persuaded to reveal its own rules or tool map, the attacker can target the agent's execution surface rather than guessing blind. That intersects directly with AI agent identity, tool authorisation, and NHI-style privilege scoping for agent actions.

Key questions

Q: What fails when an AI agent's prompt injection guardrail is bypassed by reframing?

A: The failure is usually not the model's reasoning alone, but a guardrail that treats one blocked string as proof of safety. If the same intent can be restated as a harmless-sounding request, the control never evaluates the underlying objective. Practitioners should assume attackers will probe for semantic equivalents, not repeat the first payload verbatim.

Q: Why does system prompt disclosure increase the risk of AI agent abuse?

A: Because the prompt often reveals the agent's hidden rules, tool names, and session context, which removes the guesswork from follow-on abuse. Once an attacker knows the internal operating pattern, they can target the highest-value tools and craft requests around the exact conditions the agent expects. Disclosure is therefore a force multiplier, not just an information leak.

Q: What do security teams get wrong about prompt guardrails?

A: Teams often treat prompt guardrails as if they were authorisation controls, but they are only one layer of defence. A model that filters unsafe language can still execute hidden instructions inside legitimate content if tool permissions are broad. Guardrails reduce exposure, but they do not replace separate approval checks for sensitive actions.

Q: How should organisations govern external tools used by AI agents?

A: Organisations should review external tools as security inputs, not convenience features. Each tool needs ownership, approval, metadata inspection, and ongoing monitoring for hidden instructions or unexpected behaviour. If an AI agent can act on a tool, then the tool’s provenance and control status should be governed like any other sensitive integration.

Technical breakdown

Why prompt injection succeeds against instruction-following agents

Prompt injection works because the model is trying to satisfy the latest, most locally persuasive instruction unless a stronger control boundary prevents it. In agentic systems, the model may also have access to tools, retrieval, and conversation memory, which expands the blast radius of a successful reframe. A guardrail that only checks for hostile wording is brittle because the same intent can be expressed as a research request, documentation request, or debugging question. The failure is not just lexical matching. It is the absence of a reliable policy boundary between user-facing text and privileged system instructions.

Practical implication: Treat the prompt as an untrusted input channel and move protection to policy enforcement around tool use and disclosure.

Why system prompt disclosure is operationally dangerous

A system prompt is effectively the operating manual for the agent. It can expose hidden rules, tool names, formatting constraints, and sometimes session-specific metadata that helps an attacker shape future requests. Once that information is known, the attacker no longer has to explore the agent's behaviour blindly. They can target the exact tools and conditions the agent uses internally, which sharply reduces the uncertainty that defensive teams rely on. In governance terms, prompt secrecy is not a sufficient control, but prompt exposure is still a material risk amplifier because it lowers the cost of follow-on abuse.

Practical implication: Minimise sensitive instruction content in the system prompt and assume disclosure is a plausible outcome rather than an edge case.

How semantic filtering differs from blocklist-based guardrails

Semantic filtering evaluates the intent of a request, not just its keywords. That matters because a blocked prompt can be rephrased into something that sounds benign while preserving the same underlying objective. Blocklists tend to fail at that boundary because they are tuned to known strings, not to goal equivalence. Output-side controls are the second layer: they prevent the agent from disclosing system instructions, tool inventories, or other protected state even if the input classifier is fooled. The combination is stronger because it addresses both attack paths, one at ingress and one at egress.

Practical implication: Use intent-aware classification and output disclosure rules together, because either control alone leaves a gap.

Threat narrative

Attacker objective: The objective was to extract enough internal agent metadata to bypass blind probing and target the assistant's tool surface with much higher precision.

1. Entry occurred through an authenticated user session against an internal AI assistant that allowed retrieval and tool use.
2. The first malicious request was blocked, but a reframed second request elicited the system prompt, tool list, rules, output format rules, and session identifiers.
3. Disclosure of the agent's operating instructions gave the attacker a map of which tools existed and how to target them in later interactions.
4. The end state was full prompt and session metadata exposure, which reduced the effort needed to attack the agent's tools and conversation context.

Breaches seen in the wild

• Gemini CLI prompt injection flaw 2025: Tracebit showed a poisoned README could make Gemini CLI run hidden commands and exfiltrate developer secrets; Google fixed it in 0.1.14.
• Gemini AI Breach — Google Calendar Prompt Injection: Gemini AI assistant prompt injection attack leaks sensitive Google Calendar data.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Prompt secrecy is not a control boundary: the article shows that a system prompt can be extracted even when a guardrail blocks the obvious request. That means the real control boundary has to sit around disclosure and action, not around the assumption that instructions stay hidden. For AI agent programmes, this is a governance failure mode, not a one-off test result.

Reframing is the core failure mode here: the bypass worked because the second request sounded reasonable enough to pass an intent check that was too narrow. That pattern matters for agentic AI security because the attacker is not breaking the model's logic so much as exploiting its helpfulness. Practitioners should treat semantic ambiguity as a standing adversarial condition, not an edge case.

Agent tool exposure turns prompt leakage into privilege risk: once the tool list and session details were revealed, the attacker could stop guessing and start selecting the highest-value tools. This is where AI agent identity intersects with NHI governance, because the agent's tool access behaves like an identity with scope, privilege, and lifecycle boundaries. Practitioners should govern the agent's tool permissions as carefully as any other privileged non-human identity.

Least-privilege prompting is now a design requirement: if the prompt contains anything materially useful to an attacker, disclosure becomes a direct route to misuse. The safest assumption is that a future adversary will get the prompt, so the architecture must tolerate that loss. This pushes AI security toward minimising privileged context and making tool access narrower than the conversation layer.

Testing has to match the reasoning depth of the attack: a single blocked payload is no longer meaningful assurance for agent security. The test must verify whether the assistant resists rephrased intent, not just explicit jailbreak language. For practitioners, this means red teaming, observability, and disclosure controls need to be evaluated as an integrated agent governance problem.

What this signals

Hidden prompt leakage is an AI governance issue, not just a jailbreak curiosity: once an assistant can be persuaded to reveal its own instructions, the security problem moves from input filtering to privilege design and disclosure control. That is why agent identity and tool scope need the same scrutiny we already apply to other privileged non-human identities.

Least-privilege prompting is the named concept this incident sharpens: the fewer attacker-useful details embedded in the prompt, the less value a disclosure bug can yield. For AI agent programmes, the design question is no longer whether a prompt can be hidden perfectly, but whether a leaked prompt would still be damaging enough to matter.

This pattern will force practitioners to test AI agents the way they test other privileged systems: by asking what happens when the first refusal is only the beginning of the attack. Teams that stop at one blocked attempt will miss the reframing path that an adaptive attacker will use next.

For practitioners

• Implement intent-aware prompt filtering Classify requests by intent and treat reworded disclosure attempts as the same attack class, even when the surface language looks benign.
• Block system prompt and tool disclosure at output Add explicit egress controls that prevent the agent from returning system instructions, tool inventories, session identifiers, or hidden rules.
• Minimise privileged context in agent prompts Remove tool maps, escalation rules, and other attacker-useful details from the prompt wherever the workflow can function without them.
• Scope agent tool access as privileged identity Treat each tool-enabled agent as a non-human identity with narrowly bounded permissions, and review its access like any other privileged account.

Key takeaways

• A reframed prompt injection can bypass a guardrail even after the first obvious attack is blocked, which makes semantic intent control more important than string matching.
• When an AI agent reveals its system prompt, tool list, and session context, the disclosure itself becomes an attack enabler because it removes the need for blind exploration.
• The practical defence is narrower prompts, output-side disclosure controls, and treating tool-enabled agents as privileged non-human identities with scoped permissions.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• System Prompt: The system prompt is the core instruction set that defines an LLM’s behaviour, boundaries, and response style during a session. When attackers influence or override it, they are not merely changing text. They are trying to change the model’s governing authority.
• Output Disclosure Control: Output disclosure control is the practice of preventing an AI system from revealing sensitive internal instructions, metadata, or privileged context even when the user asks for it. It is a boundary on what the system may return, not just on what it may accept.
• Agent Tool Specification: A description of when an AI agent may call a tool, what inputs it accepts, and what outputs or failures it can return. Clear tool specifications reduce invalid calls, improve evaluation consistency, and help teams control autonomous behavior in production workflows.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It gives practitioners a shared control model for securing privileged non-human actors and the systems they can touch.