How should security teams test whether an LLM can be manipulated into revealing sensitive information?

Testing for prompt injection resilience and sensitive-output leakage

Teams should treat this as a controlled abuse test of model boundaries, not a simple quality check. A useful test asks whether the LLM can be pushed from safe completion into disclosing secrets, policy text, hidden instructions, or other restricted content through adversarial phrasing, roleplay, multi-turn pressure, and indirect prompt injection. The point is to understand whether the model protects sensitive context when the prompt becomes manipulative, ambiguous, or layered with conflicting instructions.

For this question, the most relevant external references are the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework, because both frame the problem as a governance and control issue around model misuse, trust boundaries, and unsafe outputs. Security teams often miss that extraction testing needs to include partial success states, not only obvious full disclosure, because weak refusals and fragment leaks can still expose enough context to be useful to an attacker. In practice, many teams discover leakage only after an integration or agent workflow has already widened the model’s exposure surface.

Testing is most valuable when it includes realistic prompts that mirror how users, agents, and connected systems actually interact with the model. That means checking whether the model resists attempts to reveal hidden system instructions, internal policies, chain-of-thought proxies, connected-data snippets, or sensitive tokens that may appear in retrieved context. If the model sits behind tools or retrieval layers, the test should also cover whether untrusted content can be smuggled into the conversation and then echoed back as if it were trusted.

How the test behaves in practice

A strong test plan uses escalating scenarios. Start with direct extraction attempts, then move to oblique requests, then multi-turn coaxing, then prompt injection inside user-supplied content, and finally combined pressure where the model is asked to continue a task while also being tricked into exposing restricted material. That progression matters because a model may appear safe against blunt requests but still fail when the attacker first establishes context, then narrows the request, and then reframes the disclosure as harmless operational help.

• Check whether the model refuses to reveal sensitive content even when the request is wrapped in a legitimate-seeming workflow.
• Test whether the model leaks fragments of protected data after repeated rephrasing or implied authority.
• Verify that tool outputs, retrieved documents, and memory content do not become an easy proxy for direct disclosure.
• Record whether the model gives safe refusals consistently or only after certain trigger phrases.

Teams should also distinguish between prompt robustness and system design. A model that merely says no may still be unsafe if the surrounding application passes sensitive context into the prompt, fails to separate trust zones, or allows user content to influence hidden instructions. The right evaluation therefore measures both the model response and the application boundary that feeds it. If an LLM is used in an agentic workflow, this becomes even more important because the model may have access to actions, files, or downstream systems that expand the impact of a single successful extraction attempt.

Relevant guidance from the MITRE ATLAS adversarial AI threat matrix can help teams structure adversarial test cases around manipulation and abuse patterns rather than only generic unsafe prompting. Where the test touches broader governance of model behaviour and approval boundaries, the NIST AI RMF is the better frame than a purely offensive one. This guidance breaks down when teams test only curated demo prompts and never include multi-turn, context-rich, or retrieval-backed attack paths.

Where this testing gets tricky in real deployments

Tighter leakage testing often increases test complexity, requiring organisations to balance realism against the risk of overfitting to a small set of attack scripts. The hardest cases are usually not the obvious jailbreaks but the ambiguous ones, where the model partially complies, paraphrases restricted material, or reveals enough context to let an attacker keep probing.

One important variation is whether the target is a general-purpose LLM, a retrieval-augmented system, or an agentic assistant. Guidance is not fully settled on a single best test format for all of these, because the leakage path changes: in one case the model may be the weak point, while in another the weakness is the retrieval pipeline, memory store, or tool output. Another edge case is hidden instruction exposure. Some systems are less likely to reveal “secrets” in the narrow sense but still leak policy text, safety prompts, or internal routing instructions that help an attacker adapt the next prompt.

Teams should also be careful not to confuse harmless refusal with genuine resilience. A model that refuses under direct attack but leaks under translation, encoding tricks, nested quotations, or roleplay should still be treated as materially exposed. The practical question is not whether the model can be made to fail once, but whether it fails in ways that an attacker could repeat, automate, or scale.

Practitioner Guidance

What to prioritise: Measure the boundary between safe refusal and partial disclosure first, because fragment leaks and context leaks are often more operationally important than a complete refusal failure. Treat retrieval, memory, and tool outputs as part of the attack surface, not as separate concerns.

What to verify: Confirm that the test suite includes multi-turn manipulation, indirect injection, and repeated rephrasing, not just obvious jailbreak prompts. If a system only passes single-shot prompts, its protection is not yet trustworthy for production use.

Common mistake: Teams often evaluate the model in isolation and ignore the application path that supplies context. That misses the most realistic failure mode, where the model is prompted with sensitive data before the attacker even starts probing.

Practitioner takeaway: The most meaningful result is not whether the LLM can always say no, but whether it can resist manipulation without exposing enough context for an attacker to iterate toward disclosure.

Related resources from NHI Mgmt Group

• How should security teams test whether LLM safety controls still work after harmful generation starts?
• How should security teams test LLM fingerprinting in production AI agents?
• How should security teams govern sensitive data in LLM workflows?
• How should security teams test whether cloud recovery actually works?