Look for repeated exposure of prior conversation fragments, unexpected references to other users, token-like strings in responses, and outputs that contain data outside the current task scope. Those are signs that context isolation or output filtering is failing. Test with red-team prompts and measure blocked disclosures, not just model accuracy.
Why This Matters for Security Teams
Context leakage is not just a quality issue. In an AI system, it can expose prior prompts, hidden instructions, retrieved documents, or fragments of another user’s session. That creates confidentiality risk, can undermine legal privilege, and may reveal internal workflows or security controls. The practical concern is whether the system is isolating context correctly across users, sessions, tools, and retrieval layers.
Security teams often miss this because model outputs can look plausible while still disclosing information that should never have been reachable. The failure may sit in prompt assembly, retrieval permissions, session management, logging, or the output layer. Guidance from the NIST AI Risk Management Framework and related AI governance practices treats these as risk management issues, not just model tuning problems. For teams running agents, the problem is sharper because tool access and memory can amplify leakage into action.
In practice, many security teams encounter context leakage only after a red-team prompt has already extracted data that should have remained isolated, rather than through intentional boundary testing.
How It Works in Practice
Security teams should test for context leakage at the boundaries where information enters, persists, and exits the system. That means looking at prompt construction, retrieval-augmented generation pipelines, session storage, tool permissions, system instructions, and output filtering as one chain rather than separate issues. A model may appear safe in a single-turn test but still leak context when conversation state, retrieved chunks, or tool results are reused across turns.
The most reliable checks are adversarial and scenario-based. Red-team prompts should try to elicit prior conversation fragments, hidden system messages, other tenant data, and references to internal documents. Teams should also verify whether the model can be induced to repeat tokens, API keys, or metadata from retrieved sources. The OWASP Top 10 for Large Language Model Applications is useful here because prompt injection, insecure output handling, and sensitive information disclosure often overlap in the same failure path.
Operationally, mature testing includes:
- Session isolation checks across users, workspaces, and conversation threads
- Retrieval access tests to confirm that document permissions survive the RAG layer
- Tool-call reviews to ensure the agent cannot surface secrets it should only use transiently
- Output filtering tests for citations, snippets, and accidental disclosure of hidden prompts
- Logging reviews to make sure observability does not itself become a leakage channel
For agentic systems, the question is not only whether the model remembers context, but whether it can disclose or act on context it should never have had. The Anthropic report on the first AI-orchestrated cyber espionage campaign shows why this matters for security operations, because AI systems can be used to sequence sensitive actions when guardrails are weak. These controls tend to break down when shared memory, weak tenancy boundaries, or poorly scoped retrieval indexes are used in high-volume enterprise environments because the system conflates convenience with trust.
Common Variations and Edge Cases
Tighter context controls often increase engineering overhead, requiring organisations to balance user experience against isolation, auditability, and operational cost. There is no universal standard for every deployment, so current guidance suggests tailoring the test strategy to the system’s memory model, retrieval design, and business sensitivity.
Long-running assistants, customer support bots, and internal agent workflows create different leakage risks. A customer-facing chatbot may leak cross-session data if conversation state is reused incorrectly, while an internal copilot may expose policy text, incident notes, or credentials embedded in docs. In regulated settings, leaked context can also create privacy and records-management issues, especially where personal data or privileged material is involved.
Edge cases also include summarisation features, shared prompts, and cached retrieval results. A model can appear to answer only the current question while still embedding prior user data in a summary, citation, or follow-up suggestion. Best practice is evolving for memory-enabled agents, but one consistent rule is to test the full data path, not only the final response. The most useful validation is to ask what should never be visible if the system were truly isolated, then prove the system cannot surface it.
When systems blend multiple tenants, external tools, and long-lived memory, leakage detection gets harder because the boundary failure may be several layers away from the response itself.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF, NIST AI 600-1 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI risk governance should cover privacy, leakage, and boundary failures. | |
| OWASP Agentic AI Top 10 | Agentic systems can expose hidden context through prompts, tools, and memory. | |
| MITRE ATLAS | ATLAS helps model adversarial manipulation and extraction patterns against AI systems. | |
| NIST AI 600-1 | The GenAI profile addresses disclosure, prompt injection, and output safety concerns. | |
| NIST CSF 2.0 | PR.DS-1 | Context leakage is a data security problem across AI inputs, memory, and outputs. |
Apply GenAI-specific controls for prompt handling, output checks, and sensitive data protection.
