Should organisations use separate models for queries and documents in RAG?

Yes, when the use case justifies it and the operational overhead is acceptable. Separate query and document encoders can improve retrieval quality, but they require distinct vector indexes because embedding spaces are incompatible across models. Teams should adopt this pattern only if the added complexity produces measurable retrieval gains.

Why This Matters for Security Teams

RAG design is not just a search architecture choice. It affects whether users and downstream workflows receive the right context, whether sensitive material is retrieved appropriately, and whether the system can be governed as it changes. Separate encoders for queries and documents can improve semantic matching, but they also create more moving parts for model lifecycle, index governance, and auditability. That matters when RAG is used for policy answers, internal knowledge retrieval, or analyst assistance.

Security teams often underestimate the control impact of an apparently small model choice. Different embedding models mean different vector spaces, different re-indexing requirements, and different opportunities for drift if one side is updated without the other. Governance should therefore include provenance, access control, evaluation gates, and rollback paths. The NIST Cybersecurity Framework 2.0 is useful here because it frames this as a lifecycle risk, not a one-time tuning exercise. In practice, many teams discover the retrieval gap only after users start bypassing the RAG system rather than through intentional performance testing.

How It Works in Practice

Separate query and document models are usually paired in a dual-encoder retrieval setup. The query encoder turns a user question into one embedding, while the document encoder turns each chunk of source content into another. Because those embeddings are produced in different latent spaces, they cannot be mixed in the same index without breaking similarity search. That means the document corpus must be embedded with the document model, and the same query model must be used consistently at runtime.

Operationally, teams need to manage several dependencies:

• Model selection and versioning for both encoders
• Index rebuilds when the document model changes
• Evaluation datasets that measure recall, precision, and answer quality
• Controls for source data quality, chunking strategy, and metadata fidelity
• Monitoring for drift in retrieval quality after model updates

For AI governance, the key question is whether the performance gain is worth the added complexity. NIST’s AI risk guidance, including the AI Risk Management Framework, supports treating retrieval as part of the system’s risk profile rather than a back-end implementation detail. For adversarial analysis, MITRE ATLAS helps teams think about retrieval manipulation, prompt injection into indexed content, and poisoning of source material that the document encoder will faithfully represent. These controls tend to break down when organisations change one embedding model but leave the other side, index rebuilds, and evaluation baselines unmanaged.

Common Variations and Edge Cases

Tighter retrieval tuning often increases operational overhead, requiring organisations to balance improved relevance against re-indexing cost and governance complexity. That tradeoff is usually acceptable in high-value environments, but not every RAG deployment needs separate models. For many internal use cases, a single shared embedding model is simpler to govern and easier to test, especially when the corpus is small, the update rate is low, and retrieval accuracy is already adequate.

Best practice is evolving for agentic and regulated use cases. Where RAG feeds autonomous workflows, the quality of retrieved context can affect tool use, action selection, and policy enforcement, so OWASP guidance for LLM applications becomes relevant even if the primary issue is retrieval rather than generation. If the corpus includes regulated, confidential, or rapidly changing content, separate encoders may be justified because they can improve domain-specific recall. If the corpus is heterogeneous, uncurated, or multilingual, the gains are less predictable and there is no universal standard for this yet. In those environments, the guidance weakens when retrieval quality is hard to measure because business users judge usefulness subjectively and index drift is masked by fallback behaviour.

Related resources from NHI Mgmt Group

• When should organisations use document-level permissions in RAG?
• What should organisations monitor in AI workflows that use reasoning models?
• How should organisations separate identity proofing from access governance in decentralized identity models?
• How should organisations measure trust across AI use cases, agents, and models?