What do teams get wrong about securing retrieval augmented generation applications?

Why This Matters for Security Teams

retrieval augmented generation, or RAG, changes the security problem from “can the model answer?” to “what data can the application expose while answering?” That distinction matters because the weakest point is often not the model but the retrieval path, the surrounding authorization logic, and the metadata used to decide what gets indexed or returned. If teams assume vector search is inherently safe, they can leak sensitive records even when the final prompt looks harmless. NHI Mgmt Group’s Ultimate Guide to NHIs notes that 97% of NHIs carry excessive privileges, which is a useful reminder that overbroad machine access often becomes the real exposure path in AI systems. Security teams also miss that retrieval can cross tenant, document, or classification boundaries long before the model generates a response. In practice, many security teams discover RAG leakage only after users have already queried data they should never have been able to retrieve, rather than through intentional testing.

Effective RAG security starts with the idea that authorization must happen before retrieval, not after generation. The application should decide which documents, chunks, embeddings, or tools a user or agent can access, then constrain the retriever accordingly. That usually means combining RBAC or ABAC with metadata filters, document-level entitlements, and separate indexes for different sensitivity tiers. Teams should also treat embeddings as security-relevant data, because they can still reveal patterns, document presence, or adjacent context.

At the control level, NIST SP 800-53 Rev 5 Security and Privacy Controls is useful for mapping access control, audit logging, and information flow restrictions into concrete requirements. The operational takeaway is simple: retrieval policies need to be evaluated with the same rigor as application permissions, and they need to be enforced at the point where data is selected, not where the answer is written.

• Use metadata filtering to exclude documents by tenant, role, sensitivity, region, or lifecycle state.
• Separate retrieval paths for public, internal, and restricted content instead of relying on one shared corpus.
• Log both the retrieval decision and the final output so investigators can reconstruct what data was exposed.

These controls tend to break down in fast-moving development environments where every new data source is indexed into the same pipeline without a review of entitlement boundaries.

How It Works in Practice

In a secure RAG design, the user request first passes through an authorization layer that determines the caller’s effective data scope. That scope should inform the retriever, the metadata filters, and any chunk-ranking logic before anything is sent to the model. The aim is to prevent the system from retrieving content that the caller would not be allowed to read through a normal application path. This is especially important when the application supports search over tickets, contracts, source code, chat transcripts, or regulated records, because the retrieval engine can expose data that seems indirectly related but still sensitive.

Practitioners usually get the best results when they combine several controls:

• Pre-retrieval policy checks that validate user identity, tenant, and resource category.
• Metadata tags that mark content by classification, ownership, and allowed audiences.
• Index partitioning so highly sensitive content is physically or logically separated.
• Output filtering to catch accidental disclosure, but only as a backstop.

For deeper NHI context, the Ultimate Guide to NHIs is useful because RAG systems often depend on service accounts, API keys, and other non-human identities that need tightly scoped access. If those identities can read too broadly, the model inherits that reach. This is also where secret handling, logging, and rotation become part of the RAG threat model rather than separate infrastructure concerns.

Teams should also validate whether retrieval behavior changes across user prompts, languages, and tool chains. A query that looks safe in isolation can become risky when the system expands context, merges multiple chunks, or follows references across collections. That means test cases should include cross-document leakage, over-retrieval, prompt injection through retrieved text, and access drift after role changes. These controls tend to break down when retrieval spans multiple data stores with inconsistent tagging because the policy engine cannot reliably evaluate a unified access picture.

Common Variations and Edge Cases

Tighter retrieval controls often increase development and operations overhead, requiring organisations to balance security against search quality, latency, and content freshness. That tradeoff is real, especially when product teams want broad semantic recall while security teams want narrow entitlements.

Best practice is evolving for unstructured and semi-structured content, so there is no universal standard for exactly how fine-grained retrieval boundaries should be. Some teams enforce document-level access only, while others go down to passage or chunk level. The right choice depends on sensitivity, regulatory exposure, and how much context the model needs to answer accurately. If the application serves multiple tenants, cross-tenant isolation should be explicit, not implied by prompt logic or UI gating.

Edge cases often appear when an answer is assembled from several benign fragments that become sensitive in combination. Another common failure is assuming that a secure model endpoint makes the whole system safe, even though the vulnerability sits in the retrieval pipeline, not the model runtime. For organizations handling high-value machine access, the broader NHI problem described in Ultimate Guide to NHIs reinforces why access scope, credential hygiene, and offboarding discipline matter here as well. The practical boundary is simple: if the application cannot prove a caller is entitled to each retrieved source, the RAG flow is already too permissive.

Related resources from NHI Mgmt Group

• What do security teams get wrong about securing enterprise applications like PeopleSoft?
• What do teams get wrong about IDOR in AI applications?
• What do teams get wrong about session security in Python applications?
• What do teams get wrong about securing AI tools with OAuth?