LLM sensitive information disclosure exposes hidden data and secrets

At a glance

What this is: This is a guide to LLM sensitive information disclosure, showing how AI systems can leak confidential data through normal responses when they have access to training data, internal sources, or poorly isolated sessions.

Why it matters: It matters because AI applications are increasingly sitting inside identity, data, and development workflows, where leaked secrets, PII, or cross-user context can turn a model into an uncontrolled disclosure path.

By the numbers:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.

👉 Read StackHawk's guide to LLM sensitive information disclosure

Context

LLM sensitive information disclosure is what happens when a model reveals confidential data in a response because it was trained on, connected to, or allowed to retrieve information it should not surface. In this article, the primary issue is governance around data boundaries, not model intelligence, and that makes the primary keyword central to both AppSec and identity-adjacent AI controls.

The risk becomes more acute when LLMs are connected to RAG pipelines, MCP tools, codebases, or internal repositories. In those designs, the model may not be the root cause, but it becomes the disclosure channel if access scope, session isolation, and output filtering are not engineered together.

For identity and NHI practitioners, the intersection is direct: API keys, database credentials, service accounts, and other secrets can appear in model outputs just as they do in leaked code or misconfigured repos. That is why LLM security cannot be separated from secrets governance and least-privilege access design.

Key questions

Q: What breaks when an LLM can access too much sensitive data?

A: When an LLM has excessive access, it can reproduce confidential material in normal outputs, including credentials, internal documents, and personal data. The failure is usually not the model itself but the boundary around the model. If retrieval scope, session isolation, and response filtering are weak, disclosure becomes a predictable operational outcome.

Q: Why do LLMs complicate secrets and identity governance?

A: LLMs complicate governance because they consume data through tools, memory, and retrieval paths that behave like privileged interfaces. If service accounts, API keys, or document connectors are not tightly scoped, the model can surface information that an ordinary user should never see. That makes identity boundaries part of AI safety.

Q: How do security teams know if an AI system is leaking context?

A: Look for repeated exposure of prior conversation fragments, unexpected references to other users, token-like strings in responses, and outputs that contain data outside the current task scope. Those are signs that context isolation or output filtering is failing. Test with red-team prompts and measure blocked disclosures, not just model accuracy.

Q: What should teams do when AI outputs reveal secrets or PII?

A: Contain the source connection, revoke or rotate any exposed credentials, and inspect the retrieval path, prompt history, and output filters that allowed the leak. Then remove the underlying data from training, embeddings, or indexed context where possible. Treat the incident as both an AI governance issue and a secrets-management event.

Technical breakdown

How LLMs leak sensitive information through retrieval and memorisation

Sensitive information disclosure usually emerges from two paths: memorisation and live retrieval. Memorisation happens when sensitive text is absorbed during training and later reproduced in a response. Retrieval-based leakage happens when the model is connected to documents, tools, or data stores and is permitted to surface more context than the user should see. The model is not “stealing” data in the traditional sense. It is producing an output from an overbroad information boundary, which is why RAG, MCP, and prompt handling need to be governed as access control surfaces, not just AI features.

Practical implication: treat every data connection to an LLM as an access path that needs scope, audit, and response controls.

Why session isolation and output filtering are the real control layers

Cross-user leakage is often a session-management failure, not a language-model failure. If one user’s context is not cleanly separated from another’s, the model can surface prior conversation fragments, document references, or role-specific details. Output filtering then becomes the last barrier, scanning responses for secrets, PII, and internal identifiers before they reach the user. This is similar to gateway controls in other security domains, but the stakes are higher because the output can look fluent and trustworthy even when it is unsafe. Good AI governance therefore combines isolation, redaction, and policy enforcement.

Practical implication: implement response scanning and tenant isolation together, because either control on its own is incomplete.

Why secrets and credentials are especially dangerous in AI coding use cases

AI coding assistants are high-risk disclosure surfaces because they sit close to repositories, CI pipelines, environment files, and configuration artefacts. If production credentials, API keys, or connection strings appear in training data or indexed context, the model can reproduce them in a helpful-looking answer. That creates a direct bridge from AI convenience to NHI exposure. The problem is not only the leak itself, but the assumption that developers will spot it before reuse. In practice, leaked secrets in AI workflows need the same severity as exposed secrets in code or chat logs.

Practical implication: monitor AI-assisted development paths for secret exposure with the same urgency you apply to repository and pipeline leaks.

NHI Mgmt Group analysis

LLM sensitive information disclosure is an access-control problem disguised as a model problem. The article shows that disclosure happens when retrieval, memory, or session context are allowed to exceed intended boundaries. That means the control failure is usually upstream of the model output, not inside the generated text. For practitioners, the lesson is to govern data access to AI systems as tightly as any other privileged interface.

Context leakage is the named failure mode teams should watch for. When one user’s prompt history, retrieved document, or tool output can influence another user’s answer, the model becomes a cross-session disclosure channel. This is especially relevant where LLMs connect to internal repositories, support tools, or HR and finance data. The practitioner conclusion is simple: if context is shared, disclosure is already possible.

Secrets exposure through AI workflows is an NHI governance issue, not just an AppSec issue. API keys, database credentials, and service account material can be surfaced by coding copilots, document assistants, or internal chat systems if the surrounding controls are weak. That makes secrets lifecycle management part of AI risk management. The field should stop treating AI disclosures as edge cases and start treating them as predictable outcomes of overbroad identity and data permissions.

OWASP LLM02 matters because it sits at the intersection of AI governance and identity boundaries. The model can only reveal what it can reach, so the security question is who or what is authorised to retrieve that data in the first place. That is where IAM, secrets management, and policy enforcement join AI safety controls. For practitioners, the governance model must cover both the model and the identities behind its tools.

Output safety must be measured as an operational control, not a design aspiration. Organisations often say they can manage secrets and private data safely, yet disclosure still occurs through prompts, retrieval, and post-processing. This makes monitoring, redaction, and auditability essential evidence of control effectiveness. The conclusion for security leaders is to test AI outputs the way they test other privileged channels, with repeatable validation and clear escalation paths.

What this signals

Context leakage will become a recurring control theme as organisations connect LLMs to more internal data sources. The practical issue is not whether a model can generate unsafe text, but whether identity-bound data can move through retrieval paths, caches, and memory without being exposed. Security teams should expect AI review to merge with secrets governance, because the same leakage patterns will keep appearing wherever context is over-shared.

AI programmes that rely on broad tool access will need stronger policy enforcement around RAG, MCP, and response filtering. That aligns directly with NIST AI Risk Management Framework governance expectations and with the operating model described in the OWASP Agentic AI Top 10. The next maturity step is not more model tuning, but tighter control over what the model can see, store, and repeat.

For practitioners

• Classify AI data sources by disclosure risk Map every dataset, repository, and tool connected to an LLM by sensitivity, owner, and permitted output class. Block broad retrieval by default, and require explicit approval for any source that contains credentials, PII, or regulated business data.
• Separate sessions and tenants at the context layer Ensure prompts, conversation memory, and retrieved documents are isolated per user, role, and tenant. Validate that no shared cache, transcript store, or retrieval index can leak context across sessions.
• Scan AI responses before they reach users Add output filtering for secrets, token-like strings, PII patterns, and internal identifiers. Treat the response gateway as a control point and log every blocked disclosure for triage and tuning.
• Apply NHI controls to AI coding assistants Inventory API keys, service accounts, and database credentials that can be reached from coding copilots or build pipelines. Rotate exposed secrets quickly and remove hardcoded credentials from prompts, repos, and generated examples.

Key takeaways

• LLM sensitive information disclosure is a boundary failure, not simply a model-output problem.
• Leaked credentials, PII, and internal context turn AI systems into disclosure channels unless retrieval, session, and output controls are enforced.
• Security teams should govern AI access paths with the same discipline they use for secrets management and privileged identity workflows.

Key terms

• Sensitive Information: Sensitive information is a higher-risk category of personal data that attracts stricter handling requirements because misuse can create greater harm. In Australia this includes biometrics, health information, political opinions, and criminal history, which means controls for collection, access, disclosure, and transfer need tighter governance than ordinary personal data.
• Session Isolation: Session isolation is the separation of one user’s prompts, memory, and retrieved context from another’s. In AI systems, weak isolation can cause cross-user leakage, especially in multi-tenant platforms or shared assistant workflows where previous data remains reachable.
• Output Filtering: Output filtering is the control layer that scans AI responses before they are released to the user. It looks for secrets, PII, internal identifiers, and policy violations, and it is essential when models can generate fluent but unsafe disclosures from privileged context.
• Retrieval-augmented Generation: Retrieval-augmented generation is a pattern where an AI model pulls external information before generating output. The security challenge is that access rules can weaken when data is chunked, embedded, cached, or reused, so source permissions may not automatically follow the content into the model's context.

What's in the full article

StackHawk's full guide covers the operational detail this post intentionally leaves for the source:

• Concrete examples of LLM02 disclosure scenarios, including training data memorisation, cross-user leakage, and prompt injection paths
• Step-by-step guidance for sanitising data before it reaches models and for filtering responses before users see them
• Implementation detail on access control, session isolation, and privacy-preserving techniques such as tokenisation and redaction
• Plugin examples and testing considerations for AppSec teams validating AI-backed applications

👉 StackHawk's full article covers the disclosure patterns, root causes, and prevention steps in more implementation detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and machine identity security. It helps practitioners connect identity controls to the broader security programmes they already run.