Information leakage in LLMs is a governance problem, not a point fix

At a glance

What this is: This is Fiddler’s analysis of how LLMs leak sensitive data through prompts, training memory, and weak environment isolation, with layered controls and monitoring as the recommended defence model.

Why it matters: It matters because AI teams, IAM leads, and security architects need to govern how identity, permissions, and data flow through LLM and agentic workflows before leakage becomes a compliance and breach problem.

By the numbers:

• A 2023 study of 36 real-world LLM applications found that approximately 86% were susceptible to prompt injection attacks.
• Fiddler says enterprises can incur approximately $260K annually at 500K traces per day from external evaluation costs, with costs rising to $520K at 1M traces per day.

👉 Read Fiddler's analysis of information leakage security for LLMs

Context

LLM information leakage is a governance failure when prompts, training data, and runtime outputs are not controlled as distinct risk surfaces. The primary problem is not just model behaviour, but weak separation between sensitive data, identity context, and inference pathways across the AI stack.

That matters for identity and access governance because LLMs and multi-agent systems often inherit the permissions of the user, the service account, or the tool chain that powers them. Once those permissions are not propagated or checked at each step, an ordinary request can turn into unauthorised access or disclosure.

In practice, this is a familiar but more dynamic version of data exposure risk: the model can reproduce sensitive content, the retrieval layer can surface it, and the response layer can leak it before conventional controls notice. The starting position described in the article is common in early-stage AI deployments, which is what makes the risk so repeatable.

Key questions

Q: How do teams reduce the risk of sensitive data leaking from LLM outputs?

A: Teams reduce leakage by adding a response inspection layer that checks generated text for secrets, regulated data, and disallowed disclosures before delivery. They should pair that with logging and policy review so suppression events are visible to security and compliance teams. Without output controls, the model can become the last mile of accidental disclosure.

Q: Why do multi-agent AI workflows increase information leakage risk?

A: Because each handoff can widen or drop the original user’s permissions. If identity and entitlement do not follow the task through retrieval, tool calls, and response generation, downstream agents may expose data that the initiating user should never see. That is why permission context must persist across the whole execution chain.

Q: What signals show that an LLM may be leaking sensitive data?

A: Watch for abnormal response length, unusual output entropy, prompt patterns that resemble injection attempts, and rising PII detection rates. These signals often appear before a full disclosure becomes obvious. Good baselining matters because leakage usually looks like subtle drift before it looks like an incident.

Q: Who is accountable when an AI agent accesses sensitive data it was not meant to use?

A: Accountability sits with the team that approved the agent, its connectors, and its policy boundaries, not with the runtime behaviour alone. Organisations need ownership for intent, permissions, monitoring, and validation so they can prove whether the agent stayed inside its approved purpose. Without that, audit and regulatory response become retrospective guesswork.

Technical breakdown

Why prompt injection bypasses normal LLM trust boundaries

Prompt injection works because the model treats input text as both data and instruction unless the surrounding system architecture creates a hard separation. In RAG and agentic pipelines, malicious instructions can arrive through user prompts, retrieved documents, emails, or web content, then override intended behaviour if the application fails to validate context at each boundary. Input validation helps, but it is only one layer because attacks can be indirect, obfuscated, or semantically framed. Output filtering matters just as much because a successful injection may still be stopped before disclosure reaches the user.

Practical implication: validate inputs, retrieved content, and generated outputs as separate control points, not one combined model guardrail.

How memorised training data becomes a leakage channel

LLMs can reproduce memorised sequences from training data because they learn statistical patterns, not explicit records. That means a model fine-tuned or trained on sensitive data may emit names, addresses, credentials, or other records when the right prompt structure triggers recall or extraction. Larger models and long training runs increase memorisation risk, especially when small datasets are reused heavily. Access control alone does not solve this problem because the leakage originates in the model’s learned state, not in a live database query. Data minimisation, sanitisation, and careful dataset segregation reduce the chance that sensitive content ever becomes recoverable.

Practical implication: treat training data curation as a security control, not a data science task, and exclude sensitive records wherever possible.

Why multi-agent handoffs need identity context

In multi-agent systems, each handoff creates a new decision point where identity and permissions can be lost, widened, or misapplied. If a restricted user’s context does not travel with the task, downstream agents may act with broader privilege than intended, which turns a coordination feature into a privilege escalation path. This is especially important when agents call tools, query data sources, or delegate subtasks across different services. RBAC helps only if the policy is enforced at every execution layer, including retrieval, tool use, and final response generation. The control problem is not just authorisation at login; it is authorisation persistence through the full execution chain.

Practical implication: propagate user identity and permission state through every agent handoff and re-authorise before each tool call.

NHI Mgmt Group analysis

LLM leakage is an identity and data-governance problem, not a model-only problem. The article correctly shows that prompts, training corpora, and agent handoffs all create separate disclosure paths. In identity terms, the control failure is allowing runtime systems to inherit broad data access without re-checking the user’s entitlement at each step. Practitioners should treat access context as a first-class control plane concern.

Information leakage expands fastest where agentic workflows blur the line between user rights and system rights. Once a model can retrieve data, call tools, and pass work to another agent, the security question becomes whether permissions are preserved, narrowed, or lost in transit. That is a classic governance gap for IAM and PAM programmes because standing access assumptions are too coarse for delegated AI execution. The practitioner conclusion is that authorisation must remain dynamic across the full chain.

System prompt leakage should be read as a symptom of architectural overexposure. If a prompt contains secrets, URLs, or operational logic that should never reach the model layer, the problem is not disclosure detection alone but poor separation of secrets from instructions. This is where NHI governance intersects directly with AI security, because API keys, tokens, and service credentials are non-human identities with their own lifecycle. The practical response is to keep secrets out of prompts and out of reusable templates.

Continuous observability is now part of the control stack for AI governance. The article’s emphasis on output entropy, response length, and PII detection is directionally right because leakage often appears first as an anomaly, not an obvious breach. That makes detection engineering, auditability, and traceability part of the governance model rather than post-incident housekeeping. Teams should expect compliance and SOC workflows to absorb AI telemetry alongside standard application logs.

What this signals

AI leakage will force identity teams to extend control thinking from authentication into execution governance. Once a model can carry user context into retrieval, tool use, and agent handoffs, the old assumption that access is checked once at login no longer holds. Programmes that already struggle with service accounts and delegated access will need the same discipline applied to AI workflows, not just human sessions.

Context propagation is becoming the new governance boundary for agentic systems. The operational question is no longer whether a model can answer, but whether it can do so without inheriting broader rights than the user should have. That is where identity, NHI, and data governance meet, because every handoff is now a policy decision as well as a technical one.

The strongest programmes will treat leakage telemetry as part of the security control stack, not an after-action report. Baselines for output size, entropy, and PII appearance should feed SOC and GRC workflows just like any other anomalous access pattern, and the reference point for that discipline is the NIST SP 800-53 Rev 5 Security and Privacy Controls model for monitoring, access control, and auditability.

For practitioners

• Separate prompts from secrets Keep API keys, database credentials, and service tokens out of system prompts and template text. Use a vault-backed retrieval layer so credentials are injected at runtime only when the workload actually needs them.
• Enforce permission checks at every agent boundary Re-authorise retrieval, tool execution, and response generation separately, and ensure user identity travels through multi-agent handoffs. A single login event is not enough to govern downstream access.
• Sanitise training and evaluation datasets Remove or tokenise PII, PHI, and customer records before fine-tuning or evaluation. Maintain strict isolation between training, test, and production data so leakage cannot cross environments.
• Monitor for leakage-specific anomalies Track output entropy shifts, response length spikes, and PII detection rates alongside normal application telemetry. Tune alerting so unusual disclosures are caught before they reach end users.
• Build distributed audit trails for AI workflows Record who accessed what data, through which model, with which tool calls and handoffs, so you can reconstruct leakage paths and support compliance reviews.

Key takeaways

• LLM information leakage is best understood as a lifecycle control failure across prompts, data, and execution paths.
• The article’s own evidence shows why partial defences fail, because prompt injection, memorisation, and isolation gaps exploit different weaknesses.
• Practitioners need permission propagation, secrets separation, and leakage-specific telemetry before AI workflows scale further.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Model Memorization: Model memorization is when an AI system retains fragments of training data closely enough to reproduce them later. It matters because the system can surface sensitive or proprietary text through ordinary prompts, even when no one intended to expose it. The risk increases when unique records, secrets, or internal documents are used in fine-tuning.
• Context propagation: The passing of prior conversation, data, or state between tools or sessions so an actor can make use of it later. In agentic environments, context can become an access multiplier if it helps the system infer, trigger, or justify actions beyond the original request.
• Leakage Telemetry: Leakage telemetry is the set of signals used to detect sensitive data exposure in AI systems, including output anomalies, PII matches, prompt similarity, and traceability data. It turns information leakage into something observable, measurable, and suitable for security operations and compliance review.

What's in the full article

Fiddler's full blog covers the operational detail this post intentionally leaves for the source:

• Concrete prompt validation patterns for injection detection and response blocking
• Examples of output filtering logic for PII detection, system prompt fragments, and entropy anomalies
• Implementation detail for secrets vault separation and runtime credential injection
• Traceability and SIEM integration patterns for distributed audit trails across agent handoffs

👉 The full Fiddler post covers prompt validation, output filtering, and observability implementation details.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and identity lifecycle control. It helps security practitioners connect identity discipline to AI and broader enterprise risk.