Input Leakage

Input leakage is the unintended exposure of data that was supplied to a model, including prompts, hidden instructions, or sensitive context. It becomes a governance issue when applications pass secrets or privileged information into the model and assume the model will not reproduce them.

Expanded Definition

Input leakage is broader than simple data loss because the exposure can occur through normal model interaction, not only through a breach of storage or transport. In practice, it includes user prompts, hidden system instructions, retrieved context, tool inputs, and any embedded NIST AI Risk Management Framework context that a model can surface in its outputs, logs, or downstream integrations. Definitions vary across vendors on whether leakage must be direct, reconstructable, or merely inferable, but the security concern is the same: sensitive input should not become observable outside its intended trust boundary.

For NHIMG, the important distinction is that input leakage is not limited to malicious exfiltration. It can happen when an AI application is designed to helpfully echo context, summarize internal material, or preserve conversation state without adequate redaction, privilege separation, or retention controls. In agentic systems, the risk increases because the agent may forward secrets to tools, connectors, or memory layers that were never meant to receive them. The most common misapplication is treating prompt content as disposable because it was only “meant for the model,” which occurs when teams fail to recognise that model inputs may be stored, logged, or reproduced elsewhere.

Examples and Use Cases

Implementing input handling rigorously often introduces friction, because strong redaction and filtering can reduce model usefulness and complicate debugging, requiring organisations to weigh response quality against exposure risk.

• A support chatbot receives a user account number, password reset token, and internal notes; the model later repeats part of that context in a response or transcript.
• An enterprise copilot is given a privileged incident ticket containing API keys or hostnames, and those details appear in a generated summary or analytics log.
• An agentic workflow sends full browser session context to a tool chain, causing secrets embedded in prompts or headers to be propagated to external services. Guidance from Anthropic — first AI-orchestrated cyber espionage campaign report shows how model-driven workflows can be abused when sensitive context is available to the system.
• A retrieval-augmented generation application retrieves internal policy documents and returns excerpts that reveal confidential process details to users outside the intended audience.
• A developer testing a model pastes production credentials into a prompt for troubleshooting, then discovers those values in conversation history, telemetry, or shared support exports.

Why It Matters for Security Teams

Input leakage turns an AI interface into an unplanned disclosure channel, especially when teams assume the model boundary is the same as the application boundary. That assumption breaks down when prompts, hidden instructions, retrieval context, and tool arguments are retained, replayed, or surfaced to other users. Security teams need to treat model inputs as governed data because the risk is not only accidental exposure but also privilege misuse, over-retention, and cross-user contamination. This matters directly for NHI and agentic AI governance, where secrets, tokens, and delegated permissions are often passed into workflows that were never designed for secrecy. NIST’s AI governance guidance and OWASP Top 10 for LLM Applications both reinforce that input handling, data minimisation, and output monitoring are core controls, not optional hardening steps.

Once input leakage is observed in logs, chat history, or third-party integrations, containment becomes difficult because the exposure has already propagated across systems. Organisations typically encounter the operational impact only after a prompt, transcript, or agent action reveals something it should never have known, at which point input leakage becomes unavoidable to investigate and remediate.

Related resources from NHI Mgmt Group

• How can organisations reduce secret leakage in ServiceNow at scale?
• What is the difference between source control leakage and SharePoint secret exposure?
• How should security teams reduce secrets leakage without slowing developers down?
• Why do AI agents create more leakage risk than traditional applications?