OWASP LLM Top 10 LLM02 Sensitive Information Disclosure Guide: Keeping Secrets Out of Model Outputs

A large language model will repeat whatever it can see, so any personal data, credential or confidential record that reaches its training set, its context window or its logs can end up in an answer to the wrong person. LLM02:2025 Sensitive Information Disclosure is the OWASP Top 10 for LLM Applications entry that covers this risk, and it sits second on the 2025 list. This guide goes deeper than the overview: it maps the data paths that cause disclosure, sets out the controls OWASP lists and how to build them, and explains how to test, evidence and measure the result. It is written for application security engineers, AI platform teams, data protection leads and assessors who need to sign off an LLM feature before it reaches users.

Key takeaways

• Disclosure is a data flow problem, not a model problem. Most leaks come from what the application feeds the model at runtime (retrieved documents, tool results, conversation history), not from the model weights themselves.
• Prompt instructions are not a control. OWASP notes that restrictions in the system prompt about what data to return "may not always be honored and could be bypassed via prompt injection or other methods".
• Authorise before retrieval. The model should only ever see data the current user is already entitled to read, checked by the application against the user's identity, not the service account's.
• Training and fine-tuning data needs its own controls. Research has extracted gigabytes of training data from production models, and alignment did not remove memorisation, so scrub data before it is trained on.
• Logs, caches and provider retention are part of the attack surface. Prompts and completions are stored by observability tools, vector stores and model providers, and each store needs classification, access control and a retention limit.

What it is and who it applies to

OWASP defines the scope broadly: personal identifiable information, financial details, health records, confidential business data, security credentials and legal documents, plus proprietary algorithms and training methods. The risk applies to anyone who builds, buys or operates an LLM application, and it has two directions. Outbound, the model reveals something to a user or attacker who should not have it. Inbound, users hand sensitive data to a model or provider that stores, logs or trains on it, which is how the 2023 Samsung case unfolded when staff pasted internal source code into ChatGPT and the company then restricted generative AI tools.

The OWASP entry gives three examples of vulnerability:

• PII leakage: personal data is disclosed during an interaction, for example another customer's details in a support chatbot answer.
• Proprietary algorithm exposure: outputs reveal training data or model logic, enabling inversion or extraction. OWASP cites the Proof Pudding research (CVE-2019-20634), where disclosed model scores let researchers build a copy of an email filter and bypass it.
• Sensitive business data disclosure: responses include confidential plans, contracts, pricing or internal discussion.

LLM02 overlaps with three neighbours. LLM01 Prompt Injection is often the trigger, LLM07 System Prompt Leakage covers secrets placed in instructions, and LLM08 Vector and Embedding Weaknesses covers retrieval stores that ignore permissions. Treat LLM02 as the outcome those risks produce. See the OWASP Top 10 for LLM Applications Guide for the full list and the LLM Security Guide for the wider control set.

Structure and key requirements

OWASP groups its prevention strategies into six families. The table restates each one and adds what it means in engineering terms.

Where sensitive data leaks

Map every path before choosing controls. The paths below account for the incidents and research OWASP and MITRE ATLAS describe.

The last row matters more than it looks. In EchoLeak (CVE-2025-32711), a crafted email placed in Microsoft 365 Copilot's context caused it to emit a reference-style Markdown image that sent internal data out through trusted Microsoft domains. In ForcedLeak, an Agentforce agent sent CRM data to an expired domain that was still on the content security policy allowlist. Both were disclosure through output rendering, triggered by indirect prompt injection. See the Prompt Injection Guide for the trigger side.

Identity and NHI requirements

Most LLM02 failures are identity failures in disguise. The assistant calls a search index, a mailbox or a database with a service principal or API key that can read far more than the person asking the question.

• Propagate the user's identity: pass the end user's identity (an on-behalf-of token or a signed user context) to every retrieval and tool call, and enforce document and row level permissions there. See the Permission-Aware RAG Guide.
• Scope the workload identity: the application's own non-human identity should be read-only, limited to named indexes or tables, and unable to enumerate other tenants. Separate identities for ingestion (write) and query (read).
• Keep secrets out of the context window: API keys, connection strings and tokens must never be placed in prompts, tool results or memory, because anything the model sees it can repeat. Credentials belong in a secrets manager used by the application code. See the Secrets Management Guide.
• Agents act with delegated, bounded rights: where an agent fetches data, its token should carry the user's scopes and a short lifetime. See the AI Agent Authorisation Guide.
• Protect provider keys: a stolen LLM API key exposes stored conversations, files and vector stores held by the provider, not only compute. See the LLMjacking Guide.

Implementation steps

1. Inventory and classify. List every LLM feature, its data sources, tools, memory, logs and provider. Tag each source with its classification (public, internal, confidential, restricted, personal data). See the Data Classification Guide.
2. Draw the data flow. For each feature, record where data enters the context, where outputs go (screen, email, ticket, API) and where prompts and completions are stored. This is the artefact assessors will ask for first.
3. Minimise before you protect. Remove sources the use case does not need. Restrict connectors to specific sites, mailboxes or tables. Return fields, not whole records, from tools.
4. Enforce authorisation at retrieval. Filter search results and tool outputs by the requesting user's entitlements in code. Test with two accounts of different rights before anything else ships.
5. Add redaction and tokenisation. Detect PII, payment data and secrets in inputs, retrieved context and outputs using pattern and named-entity detection. Mask or tokenise in the prompt and re-identify only for authorised users. Run detection in audit mode for two to four weeks first to measure false positives, then switch to block for restricted classes. See the Tokenisation and Data Masking Guide.
6. Control output channels. Strip or proxy external images and links, allowlist rendering domains and review that allowlist for expired domains. Apply output filtering (NIST SP 800-53 SI-15) before content leaves the application.
7. Govern training and fine-tuning. Scrub datasets, deduplicate repeated records (repetition increases memorisation), document provenance and consider differential privacy for personal data.
8. Set provider and logging retention. Confirm whether the provider trains on API data and how long it keeps it. For example, OpenAI documents that API data is not used for training by default, abuse monitoring logs are kept for up to 30 days, the Responses API stores data for 30 days by default, and objects such as files, vector stores and conversations persist until deleted. Apply the same scrutiny to your own trace store.
9. Cover the inbound direction. Publish acceptable use rules and use endpoint or browser DLP to stop restricted data being pasted into unapproved AI services. See the Generative AI Data Loss Prevention Guide.
10. Roll out in rings and keep a rollback. Pilot with an internal group on non-production data, then a limited production group. Keep a feature flag that disconnects individual data sources or disables the feature without a redeploy, so a leak can be stopped in minutes.

Choosing between approaches

• Redaction versus tokenisation: redaction is simpler and suits analytics and summarisation where the value is not needed; tokenisation suits workflows where an authorised user must see the real value in the final answer.
• Pattern matching versus classifier: regular expressions catch structured data (card numbers, keys with known prefixes) with low false positives; named-entity models catch names and addresses but miss context. Use both.
• Differential privacy: worth the accuracy cost when you train or fine-tune on personal records; rarely relevant if you only use a hosted model with retrieval.
• Small versus large organisations: a small team gets most of the benefit from source minimisation, per-user retrieval and provider settings. Large organisations add a central AI gateway for redaction and logging, and label-aware controls such as Microsoft Purview sensitivity labels, where encrypted content needs VIEW and EXTRACT rights before an AI app can return it. See the AI Gateway and LLM Gateway Guide.

Evidence and assessment

Assessors and auditors need evidence that controls work, not that a policy exists. Build these tests into release gates and repeat them after any change to sources, prompts or models.

• Cross-user tests: with two test identities of different entitlements, ask for documents only one can see. Any answer that reveals the restricted content fails the release.
• Seeded canaries: place unique, fake PII and fake credential strings in test sources and training sets, then probe for them. Detection of a canary in output or logs is a definitive leak signal.
• Extraction and membership probes: for models you train or fine-tune, test whether training records can be reproduced verbatim. NIST AI 600-1 action MS-2.10-001 calls for red-teaming of training data output and membership inference. Nasr, Carlini and colleagues showed a divergence attack that made an aligned production chatbot emit training data at 150 times its normal rate.
• Indirect injection tests: plant instructions in documents, emails and form fields that the assistant will read and check whether data leaves through links, images or tool calls. See the AI Red Teaming Guide.
• Log review: sample stored prompts and completions for unredacted sensitive data, and confirm retention jobs actually delete.

Detection and metrics

Log the user identity, data sources retrieved, documents returned, redaction events, blocked outputs and outbound URLs for each interaction (NIST SP 800-53 AU-2, SI-4). Alert on canary hits, on output DLP matches for restricted classes and on retrieval of documents outside the user's normal scope. See the LLM Observability Guide.

• Cross-user test pass rate: target 100 per cent before release.
• Redaction coverage: share of prompts and outputs passed through detection, by feature.
• Restricted-data output events: count per 10,000 interactions, trending down.
• Over-privileged connectors: number of LLM data connectors using identities broader than the use case needs.
• Retention compliance: percentage of LLM stores (traces, caches, vector stores, provider objects) with an enforced retention limit.
• Time to disable a source: minutes from leak detection to disconnection.

Mapping to other frameworks

For personal data, LLM02 also engages data protection law: purpose limitation, minimisation and security of processing under the GDPR, for example. The EDPB Support Pool of Experts report on LLM privacy risks (April 2025) gives a risk methodology for LLM systems. See the AI and Privacy Guide.

Common pitfalls

• Relying on "do not reveal" instructions. A system prompt that tells the model not to share personal data is bypassed by prompt injection and by plain rephrasing. OWASP says such restrictions may not be honoured.
• One service account for every user. Enterprise search assistants that index with an administrator identity and answer without per-user filtering surface salary files and HR cases to anyone who asks the right question.
• Trusting rendering allowlists. Both EchoLeak and ForcedLeak moved data out through domains the platform trusted, one of them expired and re-registered for about five dollars.
• Logging everything forever. Observability tools capture full prompts and completions, which become the largest unclassified store of sensitive data in the estate.
• Fine-tuning on raw support data. Tickets and chat transcripts contain names, account numbers and pasted credentials. Memorised records can be extracted later, and alignment does not remove memorisation.
• Ignoring the inbound direction. Staff pasting code or contracts into public assistants, as in the Samsung case, is disclosure to a third party before any model answers. See the Shadow AI Discovery Guide.

Practitioner checklist

• Inventory every LLM feature with its data sources, tools, memory, logs and provider, and classify each source.
• Remove data sources and fields the use case does not need.
• Enforce per-user authorisation at retrieval and in every tool, and test it with two identities of different rights.
• Give the application's workload identity read-only, narrowly scoped access, with separate ingestion and query identities.
• Keep credentials out of prompts, context and memory; fetch them from a secrets manager in application code.
• Detect and redact or tokenise PII, payment data and secrets in inputs, retrieved context and outputs, starting in audit mode.
• Strip or proxy external links and images in output and review rendering allowlists for expired domains.
• Scrub, deduplicate and document every training and fine-tuning dataset; probe for memorised records with canaries.
• Set and verify retention for traces, caches, vector stores and provider-side objects; confirm provider training settings in contract and configuration.
• Keep a feature flag that disconnects a data source or disables the feature without a redeploy.

Standards and references

• OWASP: LLM02:2025 Sensitive Information Disclosure
• OWASP: Top 10 for LLM Applications 2025
• MITRE ATLAS: LLM Data Leakage
• NIST: AI 600-1 Generative Artificial Intelligence Profile
• NIST: SP 800-53 Rev. 5
• EDPB: AI Privacy Risks and Mitigations, Large Language Models
• Nasr, Carlini et al.: Scalable Extraction of Training Data from (Production) Language Models
• OWASP: LLM Prompt Injection Prevention Cheat Sheet

Related NHI Mgmt Group resources: OWASP Top 10 for LLM Applications Guide · OWASP LLM Top 10 LLM07 System Prompt Leakage Guide · LLM Security Guide · Prompt Injection Guide · RAG Security Guide