Topic starter
19/08/2026 3:28 pm
TL;DR: Sensitive data can leak from LLMs through prompt injection, memorized training data, and poorly isolated test data, according to Fiddler. The key security shift is treating leakage as a full-lifecycle governance issue, not a single-model defect, and layered controls plus continuous monitoring are needed to catch exposure in time.
Editorial analysis by NHI Mgmt Group, based on content published by Fiddler: “Information Leakage Security Optimization Model for LLMs”.
Key questions
Q: What breaks when LLM information leakage controls are not in place?
A: The control failure is usually broader than a single bad output.
Q: Why do prompt injection and memorised training data create different risk paths?
A: Prompt injection is an active attempt to override instructions at runtime, while memorised training data is exposed because the model has already absorbed sensitive patterns during training.
Q: What are the signs that an LLM is failing to protect sensitive information?
A: Warning signs include the model returning customer records, credentials, SSNs, or other confidential text that should never appear in responses.
Practitioner guidance
- Validate prompt and retrieval boundaries Classify inbound content before it reaches the model, and block or redact instruction-like material from untrusted sources, retrieved documents, and user-supplied attachments.
- Propagate user permissions through agent handoffs Carry the requester’s identity and authorisation context through every tool call, retrieval step, and agent-to-agent delegation so privilege does not expand in transit.
- Remove sensitive data before training Apply data minimisation, tokenisation, and synthetic replacement to training and evaluation sets so memorised outputs have less sensitive material to reproduce.
Bottom line: LLM information leakage is usually the result of multiple weak points acting together, not one isolated model flaw.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
19/08/2026 4:12 pm
Information leakage in LLMs is a lifecycle governance failure, not a model defect. The article shows that prompt injection, memorisation, and test-data leakage all expose the same underlying weakness: controls are being applied too late. Once sensitive data enters training, retrieval, or agent context, the model can reproduce it in ways conventional DLP and review processes do not reliably catch. Practitioners should govern data exposure from ingestion to inference, not treat leakage as a point fix.
A few things that frame the scale:
- 43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases, according to the State of Secrets in AppSec.
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: How should security teams manage permissions for AI agents?
A: Security teams should regularly assess and update the permissions granted to AI agents to ensure they align with their intended scope. Implementing a governance framework that details access levels and usage policies is crucial to mitigate risks. Moreover, continuous monitoring can detect irregular permissions that may increase exposure.
👉 Read our full editorial: Information leakage in LLMs is a governance problem, not a point fix