Threat Modelling AI Agents: A Practical Guide

Threat modelling is the cheapest point at which to make an AI agent safe. Before an agent receives credentials, a two-hour workshop can show what it will read, what it can do, who it acts for and how an attacker could turn one into the other. Afterwards, the same questions have to be answered in an incident review. This guide gives security, IAM and engineering teams a practical, repeatable method for threat modelling AI agents, combining the CSA MAESTRO framework's layered view, the OWASP Top 10 for Agentic Applications and a strong focus on identity, with a worked example you can adapt.

Key takeaways

Threat model every agent before it gets credentials, and again when its tools, data, model or autonomy change.
Start from four questions: what can it read, what can it do, who does it act for, and what can reach it?
Map every identity and credential in the design; they set the blast radius of every threat you find.
Use existing frameworks as checklists (MAESTRO layers, OWASP ASI01 to ASI10, STRIDE) rather than inventing categories.
End with specific, testable controls and owners, then verify them by red teaming.

Frameworks you can draw on

CSA MAESTRO (Multi-Agent Environment, Security, Threat, Risk and Outcome), published by the Cloud Security Alliance in 2025, structures agentic AI threat analysis across seven layers: foundation models; data operations; agent frameworks; deployment and infrastructure; evaluation and observability; security and compliance (which cuts across the others); and the agent ecosystem. It also emphasises threats that cross layers.
OWASP Top 10 for Agentic Applications (2026) gives a shared list of the most important agent risks, ASI01 to ASI10.
STRIDE (spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege) remains useful for each component and data flow.
MITRE ATLAS catalogues adversary tactics and techniques against AI systems.
OWASP NHI Top 10 covers risks in the credentials the agent relies on.

You do not need all of them. A workable approach is MAESTRO layers to make sure nothing is missed, OWASP ASI as the threat checklist, and an identity map to size impact.

Before the workshop

Participants: the agent's owner, a developer who knows the design, an IAM or identity engineer, a security architect, and someone who knows the business process and data.
Inputs: the agent's purpose; architecture diagram; list of tools, MCP servers and APIs; data sources; identity and credential design; autonomy level and approval points.
Time: 90 minutes to two hours for a single agent; longer for multi-agent systems.

The method, step by step

Step 1: Define scope and purpose

Write one sentence on what the agent is for and one on what it must never do. The second sentence is often the most useful output of the whole workshop.

Step 2: Draw the system and its trust boundaries

Draw the agent, its model provider, orchestrator, memory, tools and MCP servers, the systems behind them, the users who interact with it, and any other agents. Mark trust boundaries: where data or requests cross from one owner, network or level of trust to another. Mark every point where untrusted content enters, such as email, web, documents, tickets, tool output and messages from other agents.

Step 3: Map identities and credentials

For every arrow in the diagram, record who authenticates to whom and how:

The agent's own identity and credentials.
Delegated user tokens and their scopes.
Credentials held by tools and MCP servers to backend systems.
Model provider keys.
Credentials reachable from the agent's environment (files, environment variables, cloud metadata).

Note lifetime, scope, where each credential is stored and who owns it. This map is what turns a list of threats into a list of consequences.

Step 4: Identify threats layer by layer

Walk through the MAESTRO layers and, at each, ask which OWASP ASI risks and STRIDE categories apply. Prompts that help:

Inputs: where could an attacker place instructions the agent will read? (ASI01)
Model: what happens if the model follows those instructions, or simply makes a mistake? (ASI01, ASI09)
Memory: can one user's input influence another user's session or future tasks? (ASI06)
Tools: which tools could cause harm if misused? Can untrusted input reach a shell, query or file path? (ASI02, ASI05)
Supply chain: which tools, servers, packages and models come from third parties, and how are they verified? (ASI04)
Identity: what is the worst action possible with each credential on the map? Could the agent find and use credentials it was not given? (ASI03)
Other agents: how are they authenticated, and what happens if one lies? (ASI07, ASI08, ASI10)
Observability: would you know it happened, and could you prove what the agent did? (Repudiation)

Step 5: Rate the risks

For each threat, rate likelihood (how exposed is the entry point, how easy is the attack) and impact (what the credentials involved allow, how reversible the result is, how sensitive the data). Keep ratings simple, for example high, medium and low, and agree them in the room.

Step 6: Choose controls

For each high and medium risk, choose controls, preferring those outside the model:

Reduce access: narrower scopes, task-scoped tokens, removing tools.
Add enforcement: policy checks at the tool gateway, human approval for specific actions.
Isolate: separate agents, memory or credentials for untrusted input and privileged action.
Detect and respond: specific log fields, alerts and a kill switch.

Each control needs an owner and a way to test it.

Step 7: Validate and maintain

Red team the agent against the top threats before go-live: indirect prompt injection through each input channel, attempts to escalate privilege, and attempts to exfiltrate data.
Store the threat model with the agent's registry entry.
Re-run it when tools, data sources, model, autonomy level or delegation model change.

Worked example: an IT service desk agent

Purpose: reads incoming support emails and tickets, answers common questions, and can unlock accounts and reset multi-factor authentication for employees.

Must never: change permissions, reset credentials for privileged accounts, or act on requests it cannot verify came from the account owner.

Identities on the map: the agent's service identity; a mailbox read and send permission; a ticketing MCP server token; an identity provider admin API credential scoped to account unlock and MFA reset; a model provider key.

Threat	Layer / OWASP	Impact via identity	Rating	Controls
Attacker emails the desk pretending to be an executive and asks for an MFA reset	Inputs, model / ASI01, ASI09	Agent's IdP credential can reset MFA, enabling account takeover	High	Verify requester out of band before any reset; block resets for privileged and executive accounts; human approval for all MFA resets
Hidden instructions in an email tell the agent to forward other tickets externally	Inputs / ASI01, ASI02	Mailbox send permission allows exfiltration	High	Restrict send to internal domains; separate reading untrusted email from sending; alert on external sends
IdP admin credential broader than needed	Identity / ASI03	Could change groups or roles if misused	High	Custom IdP role limited to unlock and MFA reset for standard users; short-lived token via workload identity
Malicious update to community ticketing MCP server	Ecosystem / ASI04	Server holds ticketing token and sees all tickets	Medium	Use approved, pinned server; gateway with change detection; scoped token
One user's conversation influences answers to another	Memory / ASI06	Disclosure of another user's ticket data	Medium	Per-user memory isolation; no raw input stored in long-term memory
Actions not attributable during an investigation	Observability / repudiation	Cannot show who requested a reset	Medium	Log requester, verification result, approver, action and credential ID

The pattern is typical: the most serious threats combine an untrusted input channel with a powerful credential. Reducing the credential's scope and adding a verification step removes most of the risk, whatever the model does.

Common mistakes

Modelling the model rather than the system: spending the session on jailbreaks and ignoring what the agent's credentials allow.
Leaving identities off the diagram, so impact is guessed rather than known.
Choosing controls that live only in the system prompt.
Treating the threat model as a one-off document that is never revisited.
Skipping multi-agent interactions because each agent was modelled separately.

Practitioner checklist

Threat model each agent before issuing production credentials.
Record purpose and "must never" statements.
Draw trust boundaries and every untrusted input channel.
Map every identity and credential, with scope, lifetime and storage.
Walk MAESTRO layers with OWASP ASI01 to ASI10 as the checklist.
Rate risks by likelihood and by impact through identity.
Choose controls outside the model, each with an owner and a test.
Red team before go-live and re-model on significant change.

Threat Modelling AI Agents: A Practical Workshop Guide