How should security teams test whether an LLM can be manipulated into revealing sensitive information?

Why This Matters for Security Teams

Testing an LLM for sensitive-data leakage is not just prompt fuzzing. It is a control test for whether the model can be manipulated into crossing data boundaries, ignoring refusal logic, or exposing context that should never be user-visible. Security teams should treat this as an adversarial exercise because attackers do not ask for secrets directly at first. They probe memory, system prompts, hidden instructions, and retrieval paths until the model gives them a foothold.

The practical risk is visible in cases like the DeepSeek breach and the AI LLM hijack breach, where exposed secrets and weak identity controls turned AI systems into higher-value targets. Current guidance from the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 points toward testing for misuse, but there is no universal standard for leakage testing depth yet. In practice, many security teams discover exposure only after a model has already answered the wrong question in production.

How It Works in Practice

A useful test plan starts with the attack path, not the model. Security teams should define what “sensitive information” means in the system under test: system prompts, hidden policies, retrieved documents, API keys, session tokens, PII, or internal workflow details. Then they should run progressive abuse scenarios that escalate from benign curiosity to direct extraction attempts, including prompt injection, instruction override, role confusion, memory abuse, and chained queries that try to bypass refusal behavior.

Effective tests usually include both scripted and human-led cases. Scripted cases establish repeatability, while manual probing reveals where the model improvises under pressure. Teams should record whether the model:

• Refuses consistently when asked for protected content
• Leaks partial information through paraphrase, summaries, or “helpful” restatement
• Exposes data after multi-turn coaxing or context poisoning
• Changes behavior when the same prompt is rephrased or translated
• Returns data from retrieval layers, tools, or memory that were not intended for the requester

That testing becomes stronger when it is paired with provenance and identity controls. The OWASP NHI Top 10 highlights why model-accessed credentials and downstream tool permissions matter, while the CSA MAESTRO agentic AI threat modeling framework helps teams map how an AI system can misuse connected services. For higher assurance, teams should also validate response behavior against the NIST AI 600-1 Generative AI Profile and check whether the model is protected by short-lived, least-privilege access rather than persistent secrets. These controls tend to break down when the model has broad retrieval access plus long-lived tool credentials, because leakage then becomes a data-plane problem rather than a prompt-only problem.

Common Variations and Edge Cases

Tighter leakage testing often increases cost and operational overhead, so organisations must balance depth against release speed and model availability. The tradeoff is that shallow tests can miss the exact multi-turn behaviour attackers rely on, while deep tests can become noisy if they do not distinguish harmless hallucination from real secret exposure.

Best practice is evolving for systems that use retrieval-augmented generation, memory, or multi-agent orchestration. A model may never reveal a secret directly, but it can still leak sensitive material through a downstream tool call, a summarisation step, or another agent in the chain. That is why teams should test the full workflow, not just the chat surface. Where possible, include hidden-canary values and known sentinel secrets so the test can confirm whether the model is exposing actual protected content rather than generating plausible text.

For agentic systems, the most important edge case is that the LLM may not be the only actor with execution authority. If an agent can browse, call APIs, or invoke other agents, then a “safe” model answer can still lead to unsafe system behaviour. NHIMG research such as McKinsey AI platform breach and LiteLLM PyPI package breach shows why hidden dependency paths and compromised integrations matter as much as the model prompt itself. The practical limit appears when a test environment cannot mirror the live toolchain, because that leaves the highest-risk leakage paths unexamined.

Related resources from NHI Mgmt Group

• How should security teams test whether LLM safety controls still work after harmful generation starts?
• How should security teams test LLM fingerprinting in production AI agents?
• How should security teams govern sensitive data in LLM workflows?
• How should security teams test whether cloud recovery actually works?