Organisations get started with an AI threat model by mapping the model, its data sources, users, tools and trust boundaries, then identifying where input, output and training data can be manipulated. Start with the highest-risk use cases, catalogue likely abuse paths such as prompt injection or data leakage, and assign owners for each control.
What an AI threat model should map first
An AI threat model starts with the system boundary, not the model alone. You need a concrete map of the model, the data it consumes, the people and services that can influence it, and the trust boundaries between training, retrieval, orchestration and output handling. That gives you the places where manipulation, leakage and unsafe delegation can actually occur.
The most useful first cut is to separate what the system does from what it trusts. In practice, that means documenting prompts, system instructions, retrieval sources, plugins, APIs, downstream actions and any human review step that can be bypassed or misled.
For teams working on agentic systems, a structured method such as Threat Modelling AI Agents helps turn that boundary map into an actionable model of inputs, tools and control points. If the use case is broader AI risk governance, an AI Governance Guide is a useful companion for tying the threat model to ownership, approval and lifecycle decisions.
Which abuse paths matter most at the beginning
The first threats to catalogue are the ones that can change system behaviour without changing code. Prompt injection, indirect prompt injection, retrieval poisoning, data leakage, insecure tool calls and malicious or malformed training data are all common starting points because they exploit the system’s trust in inputs.
That is why the starting point is to identify where untrusted content can cross into trusted context. A user message, a web page, a document retrieved into context, or a tool response can all become an attack path if the model or agent treats it as authoritative.
For AI systems that rely heavily on prompts, outputs and external context, LLM Security Guide is a practical reference for the most common failure modes, especially data leakage and improper output handling. For adversarial technique mapping, MITRE ATLAS adversarial AI threat matrix is useful when you want to name the techniques rather than just describe them.
How to scope the first iteration without overbuilding it
The first version should focus on the highest-risk use cases, not every experimental AI feature. Start with the workflows that can expose sensitive data, trigger external actions, influence business decisions or reach production systems. Those are the places where the impact of a failure is easiest to justify and the control expectations are clearest.
A good early threat model also assigns an owner to each control point. Someone should own prompt controls, someone should own retrieval sources, someone should own tool access, and someone should own review or escalation paths. Without that assignment, even a correct model becomes an artefact with no operational follow-through.
Where AI is deployed as an internal service or production capability, an AI Security Maturity Model can help teams decide whether they are still at inventory-and-baseline stage or ready for deeper control coverage. For organisations that want a formalised threat-modelling approach for multi-agent systems, CSA MAESTRO agentic AI threat modeling framework is a strong reference point.
Risk and Threat Considerations
AI threat models fail when teams treat the model as the only asset. The real exposure usually sits in the surrounding trust chain, especially data sources, orchestration logic, tool access and output handling. If those boundaries are not explicit, attackers can manipulate context, induce unsafe actions or leak sensitive data without ever touching the core model weights.
Failure mechanism: Untrusted input is allowed to influence trusted context, tool execution or downstream actions, so malicious content is treated as instruction, evidence or authority.
Impact: The system can reveal sensitive data, take unintended actions, corrupt decisions or propagate compromise into connected services and workflows.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST AI RMF, NIST SP 800-53 Rev 5 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI02 — Tool Misuse | AI threat models must trace unsafe tool and action paths. |
| ASI03 — Identity & Privilege Abuse | Agent threat modelling must account for delegated identity and privilege abuse. | |
| ASI06 — Memory & Context Poisoning | Prompt injection and retrieved-content poisoning directly affect AI threat models. | |
| Recommendation — Constrain tool access and validate every agent action before execution. Restrict agent privileges to the minimum needed for each task. Treat external context as untrusted and filter it before it reaches agent memory. | ||
| NIST AI RMF | Govern | AI threat modelling supports organisational governance and accountability for AI risk. |
| Recommendation — Define ownership, approval and escalation for AI risk decisions. | ||
| NIST SP 800-53 Rev 5 | SA-15 — Development Process, Standards, and Tools | Threat modelling is part of secure AI/system development and review. |
| AC-6 — Least Privilege | AI tool and connector access should be limited to the minimum necessary authority. | |
| SI-10 — Information Input Validation | Prompt and retrieval inputs need validation because they can be manipulated. | |
| Recommendation — Embed threat modelling into the development lifecycle before deployment. Limit every AI component to the minimum permissions it needs. Validate and sanitise all model inputs before they influence decisions. | ||
| NIST CSF 2.0 | GV.RM-01 — Risk Management Strategy | Starting an AI threat model is a risk-management activity tied to prioritisation. |
| Recommendation — Set AI risk prioritisation rules for the highest-impact use cases first. | ||
Practitioner Guidance
Where to start: Build the first threat model around one high-value use case and trace a single request from ingestion to output, including retrieval, memory, tools and any human approval step. That is usually enough to expose the first real control gaps.
What to verify: Confirm that every data source, tool and external connector has an owner, an access rule and a clear trust assumption. If you cannot name who can influence it and how it is validated, it belongs in the threat model as an unresolved risk.
What practitioners underestimate: The biggest early mistake is modelling prompt risk while ignoring operational delegation. In AI systems, the dangerous failure is often not that the model is wrong, but that a wrong output is allowed to become an action.
Practitioner takeaway: The fastest way to make an AI threat model useful is to anchor it in trust boundaries and action paths, then work outward from the highest-risk use case until every sensitive input, output and tool call has an owner.
Related resources from NHI Mgmt Group
- What do organisations get wrong when they secure AI only at the model layer?
- Should organisations manage employees and AI agents under the same insider threat model?
- What breaks when organisations start AI red-teaming without a clear threat model?
- What do organisations get wrong about responsible AI when they focus only on model performance?