Prompt extraction benchmarks expose how AI agents leak security facts

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Verno Labs's blog post: Prompt extraction against benchmarked AI agents revealed that security-relevant facts hidden in system prompts can be disclosed at high rates, with the offensive campaign outperforming promptfoo under identical test conditions.

Why it matters: This matters because AI agents increasingly carry identities, tool references, scope rules and internal references inside prompts, which turns prompt leakage into a governance and access-control problem rather than a purely model-behaviour issue.

By the numbers:

• VernoRed disclosed 90.3% of the security-relevant facts hidden in the targets' system prompts.
• promptfoo disclosed 73.5% of the security-relevant facts hidden in the targets' system prompts.
• VernoRed extracted 84% of the internal references, including emails, endpoints, configuration values and secrets.

Context

Prompt extraction is not just about recovering text from a model prompt. In practice, it is about whether an AI system can be induced to reveal identity facts, scope limits, tools and internal references that should have been kept outside the model boundary.

Verno Labs frames the benchmark around AI agents that resemble production assistants, including general-purpose, financial, HR, legal and trading systems. That makes the topic directly relevant to agentic AI governance, because hidden prompt content often functions like an operational control plane for the agent.

Key questions

Q: How should security teams reduce prompt injection risk in AI agents?

A: Security teams should reduce prompt injection risk by constraining what enters the context window, limiting tool permissions, and separating untrusted retrieval content from privileged instructions. The practical goal is not perfect detection. It is to ensure that a successful injection cannot trigger wide data access, uncontrolled writes, or irreversible actions through a delegated identity.

Q: Why do exact tool names matter in agent security?

A: Exact tool names matter because they are often the identifier needed to invoke, chain or impersonate a capability. A paraphrased description may reveal useful context, but an attacker usually needs the real name, endpoint or reference to turn disclosure into abuse. That is why prompt leakage has to be treated as operational knowledge exposure, not just text recovery.

Q: What breaks when system prompts contain identity and tool data?

A: The boundary between policy and execution starts to collapse. If system prompts contain identity, scope, tools and internal references, a successful extraction attack can reveal the same facts defenders rely on to govern the agent. That creates a shadow control plane inside the prompt itself, which is difficult to audit and easy to leak.

Q: How do teams tell whether prompt testing is actually meaningful?

A: Prompt testing is meaningful when it measures disclosure by fact category and by exact identifier, not by broad similarity alone. If identity facts, rules, tools and internal references are scored separately, teams can see where disclosure creates real abuse potential. A single aggregate score hides the controls that matter most.

Technical breakdown

How prompt extraction benchmarks score security-relevant facts

Verno Bench scores extraction at the fact level rather than by text similarity. Facts that behave like concepts, such as rules or tool descriptions, are matched through retrieval and then judged for semantic leakage. Facts that behave like identifiers, such as tool names, email addresses, endpoints and secrets, only count when they are recovered exactly. That distinction matters because a paraphrased disclosure can still reveal operational knowledge even if the original wording never appears. The benchmark therefore measures what an attacker now knows, not how closely the output resembles the hidden prompt. See the OWASP GenAI Red Teaming Manual Guide.

Practical implication: Practitioners should score prompt leakage by exposed facts and identifiers, not by superficial text overlap alone.

Why agent system prompts become high-value disclosure targets

System prompts in agentic AI often bundle identity, scope, rules, tools and internal references into one hidden instruction set. That turns the prompt into an operational asset rather than a harmless string. If an attacker can induce disclosure, they can learn what the agent is allowed to do, which tools it can call, and which references or secrets are embedded in its working instructions. In security terms, the prompt can become a concentration point for policy, routing and tool access knowledge, which is why disclosure has governance consequences beyond model quality. See the OWASP LLM Top 10 LLM07 System Prompt Leakage Guide.

Practical implication: Separate sensitive control data from the prompt and treat prompt contents as governed operational material.

What the gap between tool descriptions and exact tool names means

The results show a recurring pattern: attackers can often get an agent to describe a tool's purpose even when they fail to recover the exact tool name. That difference is not trivial. In agentic workflows, exact tool names can be the key needed to invoke, chain or impersonate a capability, while descriptive language may still be enough to map the environment and plan follow-on attacks. The benchmark therefore exposes a familiar security problem in a new form: partial disclosure can still support abuse even when the attacker does not obtain a perfect copy of the prompt. See the LLM Security Guide.

Practical implication: Protect exact tool identifiers with the same care as secrets because partial disclosure can still enable misuse.

NHI Mgmt Group analysis

Prompt extraction is an identity and governance problem, not a model-parroting problem. When an agent discloses who it is, what it can do and what tools it knows about, the organisation has already leaked control-plane knowledge. That makes prompt content part of the identity boundary for agentic systems, especially when prompts encode scope, tool access and internal references. The practitioner conclusion is simple: if the prompt contains operational truth, it must be governed like policy.

Exact identifiers matter more than paraphrased leakage. Verno Labs's results show that getting a tool description is not the same as getting the exact tool name, endpoint or reference needed for abuse. This is a useful warning for teams that still treat prompt leakage as a language-quality issue. The real governance failure is allowing identity-bound metadata to sit inside a disclosure-prone channel. The practitioner conclusion is to separate descriptive prompt text from identifiers that drive execution.

Prompt leakage creates a new form of shadow AI governance debt: the system may look compliant on paper while quietly exposing the facts an attacker needs to route around policy. That matters for organisations deploying assistants across finance, HR, legal and trading workflows, where prompts often carry business-sensitive controls and references. The practitioner conclusion is to inventory prompt content as a governed asset, not an implementation detail.

Benchmarking disclosure by fact category is the right way to measure agent risk. Identity facts, rules, tools and internal references fail differently, so a single overall score hides the governance work that remains. The category split is especially important for AI systems that call other systems, because tool leakage and internal-reference leakage are the fastest routes from curiosity to abuse. The practitioner conclusion is to test each fact class against its own disclosure risk.

Prompt extraction should be treated as an attack path into agentic systems. The article's value is not the leaderboard itself but the evidence that attackers can harvest actionable knowledge even without seeing the system prompt. That should push security teams to review agent design, logging, tool naming and prompt composition as one governance surface. The practitioner conclusion is to shift controls left, before the prompt becomes the easiest place to steal working knowledge.

What this signals

Fact-level leakage is the governance signal that matters most. AI teams should stop treating prompt exposure as a fuzzy quality issue and start measuring which categories of facts can be disclosed under pressure. Identity facts, tool names and internal references do not fail in the same way, and the control response has to match the category that leaks.

Prompt secrecy is a brittle control when the prompt doubles as policy storage. The more an agent's hidden instructions carry real operating knowledge, the more disclosure becomes a direct route to misuse. That is why agent programmes need a design rule: keep identity-bound and tool-binding facts out of the prompt whenever possible.

Vulnerable prompt composition will keep showing up as an agentic AI risk pattern. Once an organisation scales assistants across finance, HR or legal workflows, the pressure to stuff operational context into prompts increases. The sustainable response is to move sensitive references into governed layers that can be audited, rotated and revoked without changing the model itself.

For practitioners

• Inventory prompt-bound security facts Catalogue which identity, scope, tool and internal-reference facts are stored in system prompts, then classify which of them are safe to disclose and which must remain outside the model boundary.
• Remove exact identifiers from hidden prompts Move tool names, endpoints, emails, configuration values and secrets into external policy, lookup or orchestration layers so the prompt no longer carries the information an attacker needs.
• Test prompt leakage by fact category Measure identity, rules, tools, tool details and internal references separately so you can see which disclosure class creates the most practical abuse risk.
• Harden agent attack surfaces before deployment Use blind testing against production-like assistants to see how easily they reveal operational knowledge through direct prompt injection, jailbreaks and impersonation.
• Treat prompt disclosure as an access-control issue Review where disclosed facts could enable tool invocation, impersonation or routing changes, then put those controls outside the prompt and into governed policy.

Key takeaways

• Prompt extraction in AI agents is a disclosure problem, not a simple text-recovery problem.
• Verno Labs's benchmark shows that attackers can recover high-value facts from hidden prompts even under matched test conditions.
• The practical fix is to keep identity, tool and internal-reference data out of prompts and test disclosure by fact category.

Related NHI Mgmt Group resources: OWASP LLM Top 10 LLM07 System Prompt Leakage Guide · LLM Security Guide · OWASP GenAI Red Teaming Manual Guide · AI Red Teaming and Testing Tools Buyer's Guide

Key terms

• Prompt Extraction: Prompt extraction is the act of using carefully crafted inputs to make a model reveal hidden instructions, embedded data, or connected-source content. It does not require a classic compromise path. The model may still appear to be functioning normally, which makes the disclosure harder to detect with traditional logging.
• System Prompt: The system prompt is the core instruction set that defines an LLM’s behaviour, boundaries, and response style during a session. When attackers influence or override it, they are not merely changing text. They are trying to change the model’s governing authority.
• Fact-Level Scoring: Fact-level scoring evaluates leakage by individual security-relevant facts instead of by general similarity to the original text. This approach is better for agent security because an attacker can learn something valuable without reproducing the prompt verbatim.
• Tool Identifier Leakage: Tool identifier leakage occurs when an agent reveals the exact name or reference of a tool it can call. That disclosure matters because exact identifiers often unlock invocation, chaining or impersonation in downstream attack paths, even when the rest of the prompt remains hidden.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is designed for practitioners who need to govern machine identities, secrets, and agent access with clear operating controls.