Information leakage in LLMs is a governance problem, not a point fix

At a glance

What this is: Fiddler argues that LLM information leakage is driven by multiple exposure paths, including prompt injection, memorised training data, and weak test-data isolation.

Why it matters: For IAM, NHI, and AI governance teams, the key issue is that permissions, data handling, and monitoring must follow the full agentic path, not just the model boundary.

Context

Information leakage in LLMs is a governance and control problem because sensitive data can surface through prompts, outputs, and learned behaviour even when teams believe the model is isolated. In practice, the weak point is often the handoff between training data, runtime context, and downstream logging, not the model alone.

That matters for identity and access governance because LLMs, tools, and agents can inherit data access in ways that traditional application controls did not anticipate. When user context does not travel consistently through retrieval, execution, and response layers, disclosure risk becomes a permissions problem as much as a model risk problem.

Key questions

Q: What breaks when LLM information leakage controls are not in place?

A: The control failure is usually broader than a single bad output. Without layered validation, redaction, and monitoring, sensitive data can enter the model through prompts, training data, or test sets and then reappear in responses, logs, or agent handoffs. That creates a governance failure where disclosure is discovered after the fact rather than prevented at the point of access.

Q: Why do prompt injection and memorised training data create different risk paths?

A: Prompt injection is an active attempt to override instructions at runtime, while memorised training data is exposed because the model has already absorbed sensitive patterns during training. One is a live control bypass, the other is latent data reproduction. Organisations need different safeguards for each, including input validation, data minimisation, and output filtering.

Q: What are the signs that an LLM is failing to protect sensitive information?

A: Warning signs include the model returning customer records, credentials, SSNs, or other confidential text that should never appear in responses. Other indicators are users pasting sensitive data into prompts, repeated leakage from the same workflow, and access patterns showing unauthorized users reaching protected content. These signals point to broken sanitization, validation, or authorization controls.

Q: How should security teams manage permissions for AI agents?

A: Security teams should regularly assess and update the permissions granted to AI agents to ensure they align with their intended scope. Implementing a governance framework that details access levels and usage policies is crucial to mitigate risks. Moreover, continuous monitoring can detect irregular permissions that may increase exposure.

Technical breakdown

Prompt injection turns input handling into a control boundary

Prompt injection is not just a bad user query. It is an instruction-conflict problem where untrusted input competes with system instructions, retrieved context, and policy constraints inside the same prompt window. That is why retrieval-augmented generation does not remove the risk. If a document, email, or page contains malicious instructions, the model may treat them as higher-priority context unless the application isolates and validates the source. Output filtering helps, but it is a compensating control, not a root fix.

Practical implication: treat prompt ingestion as a security boundary and validate retrieved content before it reaches the model.

Training data memorisation creates leakage without any live access breach

LLMs can reproduce memorised fragments of training data because they learn patterns, not records in the database sense. That means a model can emit names, addresses, credentials, or other sensitive values even when no user has direct access to the underlying source system. Membership inference and extraction attacks exploit this behaviour by probing what the model has seen and how confidently it responds. The risk increases when models are trained repeatedly on small or highly sensitive datasets.

Practical implication: reduce sensitive content before training and test for memorisation as part of model assurance.

Agent handoffs create identity propagation problems in multi-agent systems

Multi-agent workflows introduce a governance issue that looks familiar to IAM teams: the requesting user’s rights must persist across each agent-to-agent handoff. If identity and permissions are dropped, widened, or reinterpreted during a chain of tool calls, a constrained user can reach data intended for a more privileged role. This is a delegation and authorisation failure, not simply a logging gap. It becomes more severe when agents can act autonomously across retrieval, execution, and response steps.

Practical implication: propagate user identity and permission context through every agent handoff and tool invocation.

Threat narrative

Attacker objective: The attacker wants the model to disclose sensitive information or privileged context that should remain inaccessible to the requester.

1. Entry occurs when a malicious prompt, poisoned retrieval source, or misused evaluation dataset reaches the LLM context window.
2. Credential or data access is then abused when the model reveals system prompts, memorised records, or overexposed test data.
3. Impact follows when sensitive business logic, customer information, or internal configurations are disclosed into logs, outputs, or downstream agent workflows.

Breaches seen in the wild

• Gemini CLI prompt injection flaw 2025: Tracebit showed a poisoned README could make Gemini CLI run hidden commands and exfiltrate developer secrets; Google fixed it in 0.1.14.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Information leakage in LLMs is a lifecycle governance failure, not a model defect. The article shows that prompt injection, memorisation, and test-data leakage all expose the same underlying weakness: controls are being applied too late. Once sensitive data enters training, retrieval, or agent context, the model can reproduce it in ways conventional DLP and review processes do not reliably catch. Practitioners should govern data exposure from ingestion to inference, not treat leakage as a point fix.

Agentic AI introduces a permission propagation problem that IAM teams already know how to recognise. When user context does not survive tool calls and agent handoffs, the system quietly widens access beyond the original request. That makes authorisation state part of the AI control plane, not just a backend concern. Practitioners should treat handoff integrity as a core governance requirement, especially where agents can chain retrieval, execution, and response generation.

Prompt filtering alone is insufficient because leakage also happens through memory and observability layers. The article correctly places output filtering, entropy monitoring, and traceability alongside input validation, because each layer sees a different failure mode. A strong control stack has to detect bad inputs, suppress bad outputs, and preserve an audit trail without exposing more sensitive content in the process. Practitioners should align detection, response, and logging so the monitoring layer does not become another exposure path.

LLM information leakage is now a measurable assurance problem, not a theoretical AI risk. The combination of system prompt exposure, training-data reproduction, and multi-agent propagation creates a distinct governance footprint that belongs in AI risk management and identity controls. In practice, this means lifecycle assurance, not isolated testing, will determine whether leakage stays a contained event or becomes a repeatable enterprise pattern. Practitioners should measure leakage readiness as a governance outcome, not a model benchmark.

From our research library:

• 43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases, according to the State of Secrets in AppSec.
• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Permission-Aware RAG Guide

What this signals

Leakage control in LLMs is becoming a permissions-management problem as much as a model-safety problem. The operational lesson is that retrieval, tool use, and response generation all need access decisions that are consistent with the original requester. If identity context drops out during an agent handoff, the model can become a path to data it should never have been able to touch.

AI leak detection now has to work faster than the logging pipeline. Fiddler’s example of a response being captured minutes after the user session ended shows why delayed observability is not enough for AI systems. Practitioners need controls that intercept risky outputs in the moment, not only reconstruct them later from logs.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026. That pattern suggests the exposure problem is increasingly in the surrounding stack, including connectors, orchestration, and secrets handling. Teams should watch the infrastructure around the model as closely as the model itself.

For practitioners

• Validate prompt and retrieval boundaries Classify inbound content before it reaches the model, and block or redact instruction-like material from untrusted sources, retrieved documents, and user-supplied attachments.
• Propagate user permissions through agent handoffs Carry the requester’s identity and authorisation context through every tool call, retrieval step, and agent-to-agent delegation so privilege does not expand in transit.
• Remove sensitive data before training Apply data minimisation, tokenisation, and synthetic replacement to training and evaluation sets so memorised outputs have less sensitive material to reproduce.
• Separate secrets from prompt templates Keep API keys, connection strings, and authentication tokens in a vault, and inject them at runtime rather than embedding them in system prompts or static configurations.
• Monitor leakage indicators continuously Track output entropy shifts, PII detection rates, response length anomalies, and similarity to known injection patterns, then trigger incident response when thresholds change.

Key takeaways

• LLM information leakage is usually the result of multiple weak points acting together, not one isolated model flaw.
• The main exposure paths are prompt injection, memorised training data, and poorly isolated test data, all of which require different controls.
• Organisations need lifecycle governance, permission propagation, and continuous monitoring if they want to detect leakage before it becomes an incident.

Key terms

• Information Leak: An information leak is a disclosure flaw that reveals data the caller should not see, such as memory addresses, tokens, or internal state. In exploitation chains, leaks often do the quiet work of making a second-stage bug practical by removing randomness, exposing layout, or confirming that a target is reachable.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Model Memorization: Model memorization is when an AI system retains fragments of training data closely enough to reproduce them later. It matters because the system can surface sensitive or proprietary text through ordinary prompts, even when no one intended to expose it. The risk increases when unique records, secrets, or internal documents are used in fine-tuning.
• Agent Handoff: Agent handoff is the point at which a system passes an unresolved case from structured automation to an AI agent. The handoff should occur only after the workflow has collected context, because the agent is then extending evidence rather than inventing a case from scratch.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to connect identity controls to the wider security programme.