Threat Modelling RAG Systems: Assets, Trust Boundaries and Abuse Cases Step by Step

Retrieval augmented generation (RAG) puts untrusted documents, sensitive internal data and a model that cannot reliably tell instructions from content into the same context window, and most teams ship it without a threat model. A RAG system is not one component: it is an ingestion pipeline, an embedding model, a vector store, a retriever, an orchestrator, a language model and often tools, each with its own identities and trust assumptions. This guide gives a repeatable method for threat modelling a RAG system: how to draw it, which assets matter, where the trust boundaries are, a catalogue of abuse cases mapped to OWASP and MITRE ATLAS, and how to turn the model into tests and metrics. It is written for application security engineers, AI platform teams and architects. For the controls themselves, see the RAG Security Guide and, for document-level authorisation, the Permission-Aware RAG Guide.

Key takeaways

• Model the data path, not just the chat box. Most RAG abuse starts in ingestion: whoever can write to an indexed source can write to the model's context.
• Retrieved text is untrusted input. Indirect prompt injection works because the model treats retrieved content as instructions; draw a trust boundary at every point where external or user-editable content enters the index.
• Embeddings are sensitive data. Research has reconstructed 92% of 32-token inputs exactly from their embeddings, so a vector store needs the same classification and access control as the source documents.
• Identity decides blast radius. The retriever's service identity, the connectors' tokens and whether retrieval runs as the user or as the application determine what an attacker can reach.
• A threat model must end in tests. Each abuse case should produce a red-team test, a detection and an owner, and be revisited when a new source, tool or tenant is added.

Why RAG needs its own threat model

A conventional web application separates code from data; a RAG application deliberately mixes them. Greshake and colleagues showed in 2023 that LLM-integrated applications blur the line between data and instructions, so an attacker can plant instructions in content the application is likely to retrieve rather than typing them into the prompt. NIST AI 600-1, the Generative AI Profile published in July 2024, describes indirect prompt injection in the same terms and lists RAG as a data source organisations must account for (suggested action MP-1.1-001) and whose grounding and provenance should be verified (MS-2.5-005).

Three properties make RAG different from other applications you have modelled:

• Write access is indirect: an attacker does not need access to the application, only to something it indexes: a shared drive, a wiki, a ticketing system, a public web page or an inbound email.
• Effects are delayed and targeted: poisoned content sits in the index until a query retrieves it, possibly months later and for a specific user or topic.
• Authorisation is easy to lose: documents are split into chunks and embedded, and source permissions are often dropped on the way into the vector store, so retrieval can return content the user could never open in the source system.

Step 1: Model the system and draw the trust boundaries

Use the four questions from the OWASP Threat Modeling Cheat Sheet: what are we working on, what can go wrong, what are we going to do about it, and did we do a good enough job. Start with a data flow diagram that shows external entities, processes, data stores, data flows and, above all, trust boundaries. For RAG, draw two flows separately: the ingestion flow (sources to index) and the query flow (user to answer).

Mark a trust boundary at least at: each source that accepts external or broadly editable content; the connector's credential; the call to an external embedding or model API; the vector store's query interface; the point where retrieved text enters the prompt; and the point where model output reaches a renderer or tool. Pair the overview diagram with a detailed one for each pipeline if the system has several.

Step 2: Identify assets and the identities that touch them

For each asset, list the human and non-human identities that can read, write or delete it. In most RAG deployments the riskiest identity is the ingestion connector: an OAuth application or service account with tenant-wide read access to collaboration tools. MITRE ATLAS lists RAG Credential Harvesting (AML.T0082) because credentials pasted into internal documents get indexed and retrieved. See the Vector Database Security Guide for the store's own roles and tokens.

Step 3: Enumerate abuse cases

Apply STRIDE per element and per flow, then check coverage against the RAG-specific catalogues: OWASP LLM08:2025 Vector and Embedding Weaknesses and the MITRE ATLAS RAG techniques. The table is a starting catalogue; keep only the cases your architecture makes possible.

Two research results help when stakeholders doubt that these cases are practical. The PoisonedRAG study (Zou and colleagues, 2024 preprint) reported a 90% attack success rate by adding five crafted texts per target question to a knowledge base of millions of texts, and found the defences it evaluated insufficient. Morris and colleagues (2023) reconstructed 92% of 32-token inputs exactly from their embeddings and recovered full names from clinical notes. See the AI Data Poisoning and Backdoors Guide and the Prompt Injection Guide.

Step 4: Rate the threats and choose responses

Rate each abuse case on likelihood and impact, as NIST CSF 2.0 ID.RA-04 expects, using factors specific to RAG:

• Source writability: public web and inbound email are the highest; a curated, change-controlled knowledge base the lowest.
• Retrieval identity: retrieval filtered by the end user's own permissions limits leakage to what that user could already see; retrieval as a privileged application identity exposes everything indexed.
• Output reach: answers shown only to the asker are lower impact than answers that trigger tools, send messages or feed other systems.
• Data class: an index containing regulated personal data, credentials or source code raises impact for disclosure cases.

Small teams: a single internal assistant over a curated corpus can be modelled in a half-day workshop and controlled with source allow-listing and user-scoped retrieval. Large or multi-tenant platforms: model each pipeline separately, keep a shared abuse case library, and require a threat model update as a release gate for new sources and tools.

Step 5: Turn the model into controls, tests and detections

1. Write each abuse case as a test: a seeded document with hidden instructions, a query from a user without access to a sensitive document, an exported embedding sample, an answer that tries to render an external image. Run these before launch and on every change, as NIST AI 600-1 MS-2.7-007 suggests red-teaming for prompt injection and data poisoning.
2. Pilot with canaries: place canary documents and canary strings in restricted sources; any answer containing them to the wrong user is a failed control.
3. Log what was retrieved: for each answer record user, query, retrieved chunk IDs, source and ACL decision, in tamper-evident storage, as LLM08 recommends.
4. Detect ingestion anomalies: new sources, bursts of uploads, documents with hidden text or instruction-like phrasing, and chunks retrieved for unusually many unrelated queries.
5. Roll back the index: keep versioned snapshots and source lineage per chunk so a poisoned source can be purged and the index rebuilt from a known good point.
6. Review on change: new connector, new tenant, new tool or new model version reopens the threat model.

Metrics that show it is working: percentage of indexed sources with an owner and write-access review; percentage of chunks carrying ACL and classification metadata; canary leak rate (target zero); abuse case test pass rate per release; mean time to purge a reported poisoned document; and number of connector credentials with tenant-wide scope.

Framework mapping

Common failure patterns

• Threat modelling only the model: teams assess jailbreaks at the chat interface and never draw the ingestion flow, where poisoning and injection actually enter.
• Treating internal email and documents as trusted: anything an outsider can send or share becomes indexed content. The EchoLeak vulnerability in Microsoft 365 Copilot worked because a crafted external email was retrieved as context for ordinary questions, and data left through auto-loaded image references to allowed domains.
• Dropping permissions at chunking time: the vector store holds chunks without the source ACL, so the retriever serves restricted content to anyone who asks the right question.
• Assuming embeddings are anonymised: vectors are exported to analytics, shared with vendors or left in backups without the controls the source data has.
• Over-scoped connectors: a single tenant-wide read token feeds the index, so a leaked token or a compromised pipeline exposes everything, and credentials in documents get indexed too.
• No rebuild path: without per-chunk lineage and index snapshots, removing a poisoned source means guessing what to delete.

Lessons from incidents

EchoLeak (CVE-2025-32711) is the clearest public example of a RAG abuse chain, and every link maps to a row in the abuse case table: retrieval content crafting (the email was spread across topics so it would be retrieved often), indirect prompt injection, a bypassed classifier, and exfiltration through response rendering. Microsoft fixed it server side and reported no evidence of exploitation. The lesson for threat modellers is that each control in the chain was present but individually bypassable, so the model must assume layered failure. For a wider set of cases, see Prompt injection data theft.

Practitioner checklist

• Draw separate ingestion and query data flow diagrams with every trust boundary marked.
• List every indexed source with its owner, who can write to it and whether external content can reach it.
• Record the identity and scope of every connector, embedding and model API credential.
• Confirm retrieval is filtered by the end user's permissions, and test it with a user who lacks access.
• Classify the vector store at the level of its most sensitive source and restrict raw vector export.
• Walk the abuse case table against your design and record a response and owner for each relevant row.
• Turn each high-rated abuse case into an automated test and run it every release.
• Log retrieved chunk IDs and sources per answer in tamper-evident storage.
• Keep index snapshots and per-chunk lineage so a poisoned source can be purged and rebuilt.
• Reopen the threat model for every new source, tenant, tool or model version.

Standards and references

• OWASP: Top 10 for LLM Applications 2025
• MITRE ATLAS: RAG Poisoning
• NIST: AI 600-1 Artificial Intelligence Risk Management Framework, Generative AI Profile
• OWASP: Threat Modeling Cheat Sheet
• NIST: SP 800-53 Rev. 5
• arXiv: Greshake et al., Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• arXiv: Zou et al., PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation
• arXiv: Morris et al., Text Embeddings Reveal (Almost) As Much As Text

Related NHI Mgmt Group resources: Threat Modelling Guide · AI Agent Threat Modelling Guide · OWASP LLM Top 10 LLM08 Vector and Embedding Weaknesses Guide · Permission-Aware RAG Guide