Fact-level prompt extraction is replacing string-recovery scoring

At a glance

What this is: This is an analysis of why prompt extraction benchmarks need to measure security-relevant facts, not just textual overlap.

Why it matters: It matters because practitioners securing agentic AI need evaluation methods that reflect what an attacker can learn, especially where tools, endpoints, and internal references are exposed.

Context

Prompt extraction is not a simple text recovery problem. In agentic systems, a single prompt can hold identities, tool names, internal endpoints, scope rules, routing logic, and authentication details, which means traditional similarity scoring can miss the disclosures that matter most.

The governance gap is that many evaluation methods still optimise for lexical overlap or canary recovery, even though the operational risk sits in whether the attacker learns something security-relevant. LLM security guidance becomes more important here because prompt leakage is increasingly an agent security problem, not just a chatbot problem.

Verno Labs treats that mismatch as the central issue, and that starting position is increasingly typical for teams red-teaming production agents.

Key questions

Q: How should teams measure prompt extraction in agentic AI systems?

A: Measure prompt extraction by the security-relevant facts disclosed, not by the percentage of the prompt recovered. A small leak of an internal endpoint, tool name, or authentication reference can matter more than a large leak of harmless instructions. The right benchmark separates semantic disclosures from verbatim identifiers and scores each type with the method that matches its risk.

Q: What breaks when prompt extraction benchmarks rely on text similarity?

A: Text similarity hides the difference between harmless reconstruction and operational disclosure. An attacker can recover formatting language or persona instructions without learning anything useful, while a few leaked lines can expose tools, internal references, or access paths. Benchmarks that rely on overlap scores can therefore overstate success in low-risk cases and understate it in high-risk ones.

Q: When should teams treat a leaked prompt fragment as a security incident?

A: Treat it as a security incident when the fragment reveals a security-relevant fact, not just when it reproduces text. Internal endpoints, credentials, tool bindings, authentication cues, and escalation references deserve higher concern than ordinary style or persona instructions because they can change the attacker’s ability to operate against the agent or its connected systems.

Q: What is the difference between semantic leakage and verbatim leakage in prompt extraction?

A: Semantic leakage exposes meaning, such as a behavioural rule or tool description, even if the wording changes. Verbatim leakage exposes exact identifiers like endpoints, email addresses, version numbers, or tokens, where exact recovery matters. Good evaluations distinguish the two because they create different security outcomes and need different scoring methods.

Technical breakdown

Why lexical overlap fails for prompt extraction

Lexical overlap assumes that reproducing more of the original prompt equals a more successful attack. That breaks down when the recovered text is mostly harmless phrasing, while a few leaked lines reveal tools, internal endpoints, or authentication references. In security terms, the unit of analysis should be the disclosed fact, not the percentage of text recovered. This matters because the same extraction event can expose very different levels of operational risk depending on whether it leaks a persona instruction or a privileged reference used by the agent runtime.

Practical implication: Score prompt extraction by disclosed security facts, not by raw similarity to the original prompt.

How structured ground truth changes agent evaluation

A structured benchmark decomposes the hidden context into categories such as identity, rules, scope, tools, tool details, and internal references. Each category can require a different evaluation method, because some facts must match exactly while others can be judged semantically. That design reflects how real agent prompts are assembled and how they leak in practice. It also allows evaluators to separate disclosure of a tool capability from disclosure of a tool identifier or internal contact path, which a single document-level score cannot do reliably.

Practical implication: Label prompt content by fact type before testing extraction so your results distinguish semantic leakage from identifier leakage.

What fact-level extraction scoring looks like in practice

Fact-level scoring uses the best matching method for each disclosure type. Semantic facts such as behavioral rules or tool descriptions can be retrieved from the attack corpus and judged for equivalence even when the wording changes. Verbatim facts such as endpoints, email addresses, version numbers, and tokens need exact matching because partial disclosure can still be security-relevant. The result is a more realistic measure of what an attacker actually learned across multiple turns, rather than what portion of the prompt was reconstructed in one response.

Practical implication: Use mixed scoring logic so your benchmark treats tools, endpoints, and internal references differently from behavioral instructions.

NHI Mgmt Group analysis

Fact-level disclosure is the right security unit for agent evaluation. Prompt extraction becomes materially more dangerous when the attacker learns a tool name, internal endpoint, or authentication reference, even if most of the prompt remains unrecovered. That means evaluation teams need to stop treating prompt text as a single recoverable artefact and start measuring which facts were exposed. The practitioner conclusion is simple: benchmark the disclosure that changes attacker capability, not the wording that happened to be recovered.

Prompt extraction is an identity problem as much as a content problem. In agentic systems, the prompt often carries non-human identity attributes such as tool bindings, internal references, and execution scope, which means leakage can expose how the agent authenticates and behaves. That makes this a direct agent security guidance issue, not just a model-quality issue. The governance implication is that identity, privilege, and context should be evaluated together because the prompt increasingly acts like an access-control object.

Structured prompt ground truth creates a more credible security benchmark. When evaluators separate identity, rules, tools, and internal references, they can distinguish harmless reconstruction from operational disclosure. That improves red-teaming fidelity and gives programme owners a better view of where agent controls actually fail. The practitioner conclusion is to adopt evaluation models that reflect the real blast radius of leakage, not just the amount of text exposed.

Prompt extraction benchmarks are converging on disclosure semantics rather than text similarity. That shift will influence how teams judge agent risk, because the meaningful question is no longer whether a prompt can be copied, but whether the attacker learned enough to change the security boundary. Once that perspective takes hold, prompt security moves closer to governance of access-relevant facts. The practitioner conclusion is to align testing with the information the agent is entrusted to protect.

Structured disclosure gap: The article names the core failure mode correctly, because the benchmark problem is not missing more text but failing to classify what kind of information leaked. When a framework collapses all disclosures into one score, it hides the difference between harmless instruction recovery and exposure of internal references or tool access paths. The practitioner conclusion is to build scoring models that preserve that distinction.

What this signals

Prompt extraction will be governed by disclosure semantics, not similarity scores. As agents become more stateful and more tool-connected, the interesting question is whether an attacker learned anything that changes access or execution, not how much of the original wording came back. That pushes evaluation teams toward risk-weighted scoring, fact classification, and better handling of internal references.

Agent prompt security is becoming a control-plane issue. Once prompts carry tool schemas, internal routing logic, and authentication references, leakage starts to resemble exposure of privileged runtime metadata rather than a simple model output defect. Practitioners should expect agent assessments to move closer to identity and privilege review, especially where connected tools can execute high-impact actions.

For practitioners

• Define disclosure categories before testing Separate identities, rules, tools, tool details, scope, and internal references before you run extraction tests so each leaked fact is scored in the right risk class.
• Score security-relevant facts independently Treat a leaked endpoint, token, or internal reference as a separate control failure from a leaked persona instruction or formatting rule.
• Use different verdict logic for verbatim and semantic leaks Require exact matching for identifiers and judge semantic equivalence for behavioural rules and capabilities so partial disclosure is not undercounted.
• Red-team the full multi-turn attack path Collect responses across the whole campaign and evaluate the merged corpus, because real extraction often spreads across several small disclosures rather than one obvious leak.

Key takeaways

• Prompt extraction needs to be assessed as disclosure of security-relevant facts, because text recovery alone does not capture the real attacker value.
• The benchmark problem in agentic systems is that a small leak can expose tools, endpoints, or authentication references that matter more than most of the recovered prompt.
• Practitioners should classify leaked information by type and score verbatim and semantic disclosures differently to avoid misleading risk assessments.

Related NHI Mgmt Group resources: LLM Security Guide · SIEM Guide · CWE Top 25 Guide

Key terms

• Prompt Extraction: Prompt extraction is the act of using carefully crafted inputs to make a model reveal hidden instructions, embedded data, or connected-source content. It does not require a classic compromise path. The model may still appear to be functioning normally, which makes the disclosure harder to detect with traditional logging.
• Semantic Disclosure: Semantic disclosure is leakage where the meaning of an instruction or capability is recovered even if the exact wording is different. It matters because a model can reveal a behavioural rule, scope limit, or tool purpose without reproducing the original text verbatim, yet still give an attacker actionable intelligence.
• Verbatim Disclosure: Verbatim disclosure is the exact recovery of identifiers or sensitive strings such as endpoints, email addresses, version numbers, or tokens. These facts often require exact matching in evaluation because even partial or formatted changes can still reveal operational details that increase attacker leverage.
• Fact-Level Evaluation: Fact-level evaluation scores each security-relevant item independently instead of treating a prompt as one document. This approach is better for agentic systems because different disclosure types need different checks, and one broad similarity score can hide the real security impact of a leak.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity control to the broader security programmes they run.