What should organisations do when RAG systems can reach sensitive data?

Why This Matters for Security Teams

When RAG systems can reach sensitive data, the retrieval layer becomes part of the trust boundary, not just a convenience feature. That matters because the model may surface data it was not intended to see, and prompts, tool calls, or embeddings can become indirect paths to disclosure. NIST Cybersecurity Framework 2.0 reinforces the need to manage access and protect assets across the full lifecycle, not only at the application edge.

For security teams, the practical mistake is assuming that “read-only” is harmless. A retrieval path that can query customer records, source code, incident notes, or internal policy documents can still expose regulated or operationally sensitive information through summaries, citations, or prompt injection. The Ultimate Guide to NHIs — Key Research and Survey Results shows how often identity and secret sprawl already undermine control: 79% of organisations have experienced secrets leaks, and 97% of NHIs carry excessive privileges. That same pattern appears in RAG deployments when retrieval permissions are inherited from broad application roles instead of being designed as a separate control plane.

In practice, many security teams discover over-broad retrieval only after a model has already exposed information that was never meant to leave the source system.

How It Works in Practice

The safest pattern is to treat retrieval as a governed, least-privilege workflow. The RAG application should not query data sources with a single shared service account that can “see everything.” Instead, it should use workload identity, scoped tokens, and policy checks that determine which corpus, record class, or document label can be retrieved for a specific request. That means separating authentication of the application from authorisation of the data access path.

Operationally, teams should define retrieval scopes by dataset sensitivity, business function, and user context. For example, a support assistant may retrieve product manuals and approved troubleshooting notes, while a finance assistant may query only a narrow ledger subset. When possible, access should be short-lived and task-specific, with logging that captures the user, agent, query, source, and returned chunks. This aligns with the governance logic described in Ultimate Guide to NHIs, where visibility and rotation are core controls rather than afterthoughts.

• Apply least privilege to retrieval endpoints, not just to the application account.
• Use separate policies for read access, chunk selection, and post-retrieval transformation.
• Log every retrieval event with source, scope, and decision rationale.
• Block sensitive classes unless there is an explicit business justification and approval.
• Re-evaluate access when the prompt, user, or task context changes.

For policy design, current guidance suggests combining NIST Cybersecurity Framework 2.0 with request-time authorisation patterns and strong identity controls. Teams that also use content classification, prompt-injection filtering, and output redaction reduce exposure further, but those controls are compensating measures, not substitutes for retrieval governance. The NIST Cybersecurity Framework 2.0 is useful here because it pushes teams to identify assets, protect them proportionately, and monitor for misuse across systems and dependencies.

These controls tend to break down when retrieval is embedded inside multiple downstream tools that share one over-privileged connector, because policy decisions become opaque and impossible to enforce consistently.

Common Variations and Edge Cases

Tighter retrieval controls often increase latency, engineering overhead, and approval complexity, so organisations have to balance data minimisation against usability. That tradeoff is real, especially in high-volume support, research, or incident-response environments where users expect broad searchability.

One edge case is public-plus-sensitive corpora. Best practice is evolving, but guidance generally favours explicit segregation rather than relying on model behaviour to ignore restricted content. Another common exception is regulated data such as HR, legal, or health records, where retrieval may be permitted only for narrowly defined workflows and may require masking before chunks reach the model. If the RAG pipeline can be steered by untrusted content, prompt injection can cause the model to request or reveal adjacent data that the original user should never access. In those environments, retrieval policy must be evaluated at runtime, and the model should never become the deciding authority on what it is allowed to read.

That is why organisations should pair access control with auditability, redaction, and data domain separation. RAG can be useful, but the retrieval layer must be treated like any other sensitive identity-enabled path, not a generic search box.

Related resources from NHI Mgmt Group

• How should organisations govern MSP access to sensitive systems?
• How should security teams handle privacy rights requests when customer data is spread across multiple systems?
• Who is accountable when unsanctioned SaaS stores sensitive business data?
• What should organisations do when AI systems need production access?