Most AI red teaming still asks whether a model can be made to say something it should not, but for an agent that holds real credentials the question that matters is whether it can be made to do something it should not. An agent with mail, file, database or cloud access turns a successful prompt injection into an identity event: data read under someone else's authority, a token used outside its purpose, a privilege gained through delegation. This guide covers red teaming AI agents for identity and access abuse, including privilege escalation, credential discovery and misuse, confused deputy and delegation abuse, data exfiltration, memory and tool poisoning, approval bypass and rogue agents. It sets out how to scope, run, detect, report and repeat tests that measure real blast radius. It is written for red teams, AppSec and IAM engineers, and the security leads who own agent deployments.
Key takeaways
- Test actions and access, not only outputs. The finding that matters is what the agent read, changed or sent with its credentials, not whether it produced an unsafe sentence.
- Attack through realistic channels. Indirect prompt injection through email, documents, tickets, calendar invites, web pages, tool outputs and other agents is the main entry point, and OWASP says it is "unclear if there are fool-proof methods of prevention".
- Rate findings by impact through identity. Severity follows the scope, lifetime and reversibility of the credential the agent used, not how clever the injection was.
- Fix outside the model first. Narrower scopes, removed tools, per-action authorisation, approval gates and isolation of untrusted input close more findings than another prompt filter.
- Make it a regression suite. Re-run the attack set whenever the model, prompts, tools, connectors or permissions change, and feed results into threat models and detection rules.
Why agent red teaming is an identity exercise
NIST's Generative AI Profile (NIST AI 600-1) describes AI red-teaming as "a structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful, or discriminatory outputs". That definition centres on outputs. Agents add a second layer: the model's decision is handed to tools that act with an identity. The OWASP Top 10 for Agentic Applications for 2026 puts this in ASI03 Identity and Privilege Abuse, which "exploits dynamic trust and delegation in agents to escalate access and bypass controls", and notes that without a distinct, governed identity an agent "operates in an attribution gap that makes enforcing true least privilege impossible".
OWASP describes ASI03 as the agentic evolution of LLM06:2025 Excessive Agency, often reached through LLM01:2025 Prompt Injection. LLM06 names three root causes: excessive functionality, excessive permissions and excessive autonomy. An identity-focused red team tests all three: which tools the agent can reach, what each credential allows, and which actions run without a human or policy check.
Recent incidents show the pattern. In EchoLeak 2025, a crafted email retrieved by Microsoft 365 Copilot as context could pull data from the user's environment with no click, using the signed-in user's access. In the Gemini AI Calendar Prompt Injection 2026, Miggo Security showed a calendar invite steering Gemini to write private meeting summaries into a new event with the victim's delegated access, despite a separate detection model. Neither needed attacker credentials. See the Agentic AI Security Guide.
Scoping
Choose target agents by reach
Prioritise agents with write access, sensitive data, external communication, financial actions or the ability to delegate to other agents. A read-only internal summariser is a lower priority than an agent that can send email, issue refunds, change configuration or call production APIs.
| Priority | Agent profile | Why it comes first |
|---|---|---|
| 1 | Write or delete access to production systems, payments or identity stores | Injected instructions become irreversible actions |
| 2 | Reads untrusted external content and can send data out (email, web, tickets) | Classic injection-to-exfiltration path |
| 3 | Orchestrators and agents that delegate or hold other agents' tokens | Delegation chains spread one compromise across many identities |
| 4 | Agents with persistent memory shared across users or sessions | Poisoned memory persists and crosses user boundaries |
| 5 | Read-only, single-user agents with no outbound channel | Lower blast radius; cover in regression testing |
Environment
- Production-like, not production: use realistic identities, scopes, connectors and test data so results reflect real blast radius; avoid live customer data and irreversible production actions.
- Same credentials model: if production uses delegated user tokens or a shared service account, so should the test environment.
- Canary data and honeytokens: seed fake secrets and records so the team can prove whether the agent reached them.
Rules of engagement
Define allowed techniques, systems in scope, stop conditions, who can trigger the kill switch and who to call if the agent acts on real systems. NIST SP 800-53 control CA-8(2) Red Team Exercises frames red teaming as simulating adversary attempts "in accordance with applicable rules of engagement". Microsoft's planning guidance for LLM red teaming adds practical items: clear instructions per round, a shared place to record every input and output with an identifier for reproducibility, and someone on standby during testing.
Inputs
Bring the agent's threat model, identity and credential map (every token, its scope, lifetime and owner), tool list, connector list, memory design and approval gates. See the AI Agent Threat Modelling Guide.
Test scenarios
The table maps each scenario to OWASP Agentic Top 10 entries and to MITRE ATLAS techniques (release 2026.09). Use the IDs in test case names so findings line up with threat models and detection content. See the MITRE ATLAS Guide.
| Scenario | What to try | OWASP | MITRE ATLAS |
|---|---|---|---|
| Indirect prompt injection to action | Place instructions in content the agent reads (email, document, invite, ticket, web page) and see whether it takes an unintended action | ASI01, ASI02 | AML.T0051.001 Indirect, AML.T0053 AI Agent Tool Invocation |
| Privilege escalation | Induce the agent to request broader scopes, assume other roles or use admin tools | ASI03 | Tactic AML.TA0012 Privilege Escalation |
| Credential discovery | Lead the agent to read configuration files, environment variables, agent config or secret stores and use what it finds | ASI03 | AML.T0055 Unsecured Credentials, AML.T0083 Credentials from AI Agent Configuration, AML.T0098 AI Agent Tool Credential Harvesting |
| Confused deputy | As a low-privilege user or agent, get the agent to perform actions only its own identity may do | ASI03 | AML.T0053 AI Agent Tool Invocation |
| Delegation abuse | Get a worker agent to act with the orchestrator's full authority, or a low-privilege agent to relay instructions a high-privilege agent executes | ASI03, ASI07 | AML.T0053 |
| Authorisation drift | Reduce a user's rights mid-workflow and see whether the agent finishes with the old authorisation | ASI03 | None specific |
| Data exfiltration | Get the agent to read sensitive data and send it through another tool or channel | ASI02 | AML.T0086 Exfiltration via AI Agent Tool Invocation |
| Destructive action | Steer the agent to delete or overwrite data with a tool it legitimately holds | ASI02 | AML.T0101 Data Destruction via AI Agent Tool Invocation |
| Memory poisoning | Plant persistent instructions or cached secrets and test whether they affect later sessions or other users | ASI06, ASI03 | AML.T0080.000 Memory |
| Tool and MCP poisoning | Modify a tool description or introduce a malicious or look-alike server and observe behaviour | ASI02, ASI04 | AML.T0110 AI Agent Tool Poisoning |
| Approval bypass | Make a high-impact action skip approval, or mislead the approver with a false summary | ASI09 | None specific |
| Rogue or impersonating agent | Register an agent with a forged descriptor (for example "Admin Helper") and see whether others route privileged tasks to it | ASI03, ASI07, ASI10 | None specific |
OWASP's ASI03 entry supplies ready-made cases: un-scoped privilege inheritance, memory-based privilege retention, cross-agent trust exploitation, time-of-check to time-of-use drift in workflows and synthetic identity injection. For MCP-connected agents, add token passthrough: the MCP security best practices warn that a server forwarding tokens with the wrong audience can cause the confused deputy problem, where "the downstream API may incorrectly trust the token as if it came from the MCP server". See the MCP Security Guide and the Prompt Injection Guide.
Running the tests
- Baseline: record the agent's normal behaviour, tools, tool sequences and effective permissions. OWASP recommends a behavioural baseline covering goal state, tool-use patterns and access patterns; it doubles as your comparison point.
- Discover capabilities: ask the agent what it can do and which tools and data sources it has. ATLAS lists this as AML.T0133 Discover AI Agent Runtime Capabilities, and an attacker will do it first.
- Attempt each scenario through several input channels and phrasings. Injection success is probabilistic, so repeat each case and record the success rate, not a single pass or fail.
- Chain the steps: real attacks combine discovery, credential access and exfiltration. MAESTRO's cross-layer threats, including privilege escalation in one layer used to reach others, are a useful prompt for multi-step chains.
- Observe the whole chain: model decisions, tool calls, credentials presented, downstream API calls and the logs each system wrote.
- Check detection: did monitoring alert, and could the SOC reconstruct who asked, which agent acted and with which credential?
- Test containment: trigger the kill switch, confirm credentials are revoked and sessions end, and time it.
Evidence to record for every attempt
- Input: channel, payload, timestamp and a unique ID so the case can be replayed.
- Agent behaviour: response, tool calls made and arguments passed.
- Identity: which credential or token was used, its scope and whether it was delegated.
- Effect: data read, written, sent or deleted, and whether canary data was touched.
- Defence: what alerted, what blocked, and what the logs show.
Human testers find creative chains; automated tooling gives volume. NIST AI 600-1 notes that GAI-led red-teaming "can be more cost effective than human red-teamers alone". See the AI Red Teaming and Testing Tools Buyer's Guide.
Detection and monitoring
Every exercise also tests the blue team. Agent misuse is hard to see because, as OWASP's ASI02 tool chaining example puts it, when "every command is executed by trusted binaries under valid credentials", host-centric monitoring may see no malware or exploit. Detection has to sit on identity and tool telemetry.
| Scenario | Signal the SOC should see | Where it comes from |
|---|---|---|
| Injection to action | Tool sequence outside the baseline straight after untrusted content was read | Agent trace and tool-call logs |
| Privilege escalation, delegation abuse | An agent requesting new scopes, or a low-privilege agent handed higher-privilege scopes | Identity provider, token service, orchestration logs |
| Credential discovery | Reads of config or secret files outside the task; a token used from a new workload or for a new purpose | File and secrets-manager audit logs, API gateway |
| Exfiltration | Sensitive read followed by an outbound send to a new recipient or domain | Data access logs, email and egress logs |
| Memory poisoning | Memory writes containing instructions; behaviour change across sessions | Memory store audit |
| Rogue agent | Unregistered agent identity or descriptor receiving tasks | Agent registry, A2A logs |
OWASP's ASI03 mitigations call for monitoring "when agents request new scopes or reuse tokens outside their original, signed intent" and flagging when a low-privilege agent inherits higher-privilege scopes. Token passthrough breaks this, because downstream logs then show a different identity from the one that acted. See the AI Agent Observability and Incident Response Guide.
Reporting
For each finding, report the entry point, the chain of actions, the identity and credentials involved, the impact, whether it was detected and how long containment took. Microsoft notes that red team examples expose the risk surface and are not a replacement for systematic measurement, so do not present one success as a rate.
| Severity | Impact through identity | Example |
|---|---|---|
| Critical | Irreversible change or broad data access using a long-lived or account-wide credential, undetected | Agent uses a token found in a file to delete production data |
| High | Exfiltration or privileged action using delegated user access or an inherited scope | Injected email makes the agent send internal data to an outside address |
| Medium | Action outside the task but within narrow scope, or blocked late by an approval gate | Agent drafts an unauthorised change that a reviewer catches |
| Low | Unintended output with no tool action, or action blocked by policy and alerted | Agent discloses its tool list on request |
Recommend controls outside the model first: narrower scopes, removed tools, task-scoped and short-lived tokens, per-action authorisation through a policy engine, human approval for irreversible actions, and isolation of untrusted input. These are OWASP's ASI03 mitigations, and ATLAS lists matching mitigations such as AML.M0028 AI Agent Tools Permissions Configuration, AML.M0029 Human In-the-Loop for AI Agent Actions and AML.M0030 Restrict AI Agent Tool Invocation on Untrusted Data. See the AI Agent Authorisation Guide and the OAuth Token Exchange and Delegation Guide.
Making it continuous
- Regression suite: automate the scenario set and run it whenever the agent's model, system prompt, tools, connectors, memory design or permissions change.
- Human-led rounds: combine automation with periodic human-led exercises for creative attack chains; Microsoft suggests rotating testers across harms to keep perspectives fresh.
- Close the loop: every confirmed finding becomes a regression case, a threat model update and, where possible, a detection rule.
- Purple team the fixes: re-run the original chain with the defender watching to confirm the control and the alert both work. See the Purple Teaming Guide.
- Fit the wider programme: agent testing sits alongside penetration testing and red teaming of the rest of the estate. See the Penetration Testing Guide and the Red Teaming Guide.
NIST AI RMF subcategory MEASURE 2.7 asks that AI system security and resilience "are evaluated and documented", and OWASP's ASI01 guidance includes periodic red-team tests simulating goal override with verification of rollback.
Common failure patterns
- Testing the model, not the agent. Jailbreak benchmarks on the base model say little about what a connected agent can do with its tools. Test through the application and its real connectors.
- Clean test credentials. Test environments with tightly scoped tokens hide the over-privileged reality. In PocketOS Database Deletion 2026, an agent working in staging found an account-scoped Railway token in an unrelated file and deleted the production volume and backups.
- Treating instructions as controls. In the Replit AI Agent Database Deletion 2025, the agent deleted a live database during a declared code freeze. Red team the enforced permissions, not the prompt.
- Ignoring the tool supply chain. The postmark-mcp Malicious MCP Server 2025 blind-copied every email an agent sent to an attacker through the user's own credentials. Include third-party tools and servers in scope.
- Relying on a detection model alone. In the Gemini calendar case, a separate model to detect malicious prompts was already in place and the path still existed. Findings should drive scope and authorisation changes as well as filter tuning.
- No detection or containment check. Exercises that only record whether an attack worked miss whether anyone would have noticed or could have stopped it.
- Findings without owners. Identity fixes span IAM, platform and application teams; without a named owner and retest date they stall.
Framework mapping
Control IDs below were checked against NIST CSF 2.0, the SP 800-53 Rev. 5 catalogue, the ISO/IEC 27001:2022 Annex A list, CIS Controls v8.1 and the OWASP lists.
| Requirement | NIST CSF 2.0 | NIST SP 800-53 Rev. 5 | ISO/IEC 27001:2022 Annex A | CIS Controls v8.1 | OWASP |
|---|---|---|---|---|---|
| Red team programme with rules of engagement | ID.IM-02 | CA-8, CA-8(1), CA-8(2) | A.8.29 | 18.1 | ASI01 (periodic red-team tests) |
| Agent testing before release and on change | ID.RA-01 | SA-11, SA-11(5) | A.8.25, A.8.29 | 16.13 | LLM01, ASI01, ASI02 |
| Least privilege for agent identities | PR.AA-05 | AC-6, AC-6(5) | A.5.15, A.8.2 | 6 | ASI03, LLM06, NHI5 |
| Credential exposure findings | PR.AA-01 | IA-5 | A.5.17 | 5 | NHI2, NHI7 |
| Detection of agent misuse | DE.CM-03, DE.CM-09 | SI-4, AU-6 | A.8.15, A.8.16 | 8 | ASI03, ASI10 |
| Containment and kill switch tests | RS.MI-01 | IR-3 | A.5.26 | 17 | ASI10 |
| Remediation and retest | ID.RA-06 | CA-5 | A.8.8 | 18.3 | ASI03 |
For AI-specific governance, NIST AI RMF MEASURE 2.7 (security and resilience evaluated and documented) and NIST AI 600-1 action MS-2.7-007, which recommends AI red-teaming to assess resilience against attacks including prompt injection, sit over all rows. See the NIST AI RMF Guide.
Metrics that show progress
| Metric | What it shows | Target direction |
|---|---|---|
| Share of high-reach agents red teamed in the last cycle | Coverage of the agents that matter most | Up, towards all priority 1 to 3 agents |
| Scenario success rate per agent (attempts succeeded / attempts made) | Resistance to injection and misuse over time | Down |
| Findings rated critical or high through identity | Blast radius of the credentials agents hold | Down |
| Share of successful attacks detected by monitoring | Blue team visibility of agent misuse | Up |
| Time from kill switch to credential revocation | Containment speed | Down |
| Median time to remediate and retest a finding | Fix throughput | Down |
| Share of findings fixed by identity or authorisation controls | Whether fixes go beyond prompt filters | Up |
| Agent changes released without a regression run | Discipline of continuous testing | Down, towards zero |
Implementation roadmap
| Phase | Actions | Outcome |
|---|---|---|
| First 30 days | Inventory agents and rank by reach; agree rules of engagement and kill switch owner; build the identity and credential map for the top agents; set up a test environment with realistic scopes and canary data | Scope and safe conditions for a first exercise |
| Days 31 to 90 | Run a human-led exercise on priority 1 and 2 agents covering the scenario table; check detection and containment; report by impact through identity; assign owners | First findings with fixes outside the model |
| Days 91 to 180 | Automate the scenario set as a regression suite in the release pipeline; add delegation, memory and tool supply chain cases; turn findings into detection rules; retest fixes in purple team sessions | Repeatable testing on every significant change |
| Ongoing | Extend to priority 3 to 5 agents; track the metrics; update scenarios as OWASP and ATLAS add techniques | Coverage and resistance that improve over time |
Most findings resolve through least privilege for agent credentials. See the JIT Access Guide, the Secrets Management Guide and the Zero Trust for AI Agents Guide.
Practitioner checklist
- Rank agents by write access, sensitive data, external reach and delegation, and test the top tier first.
- Agree rules of engagement, stop conditions and a kill switch owner before testing.
- Test in a production-like environment with realistic credentials and canary data.
- Bring the threat model and a full identity and credential map to every exercise.
- Cover injection-to-action, privilege escalation, credential discovery, confused deputy, delegation, exfiltration, memory and tool poisoning, approval bypass and rogue agents.
- Repeat each case across channels and phrasings and record success rates.
- Record input, tool calls, credential used and effect for every attempt.
- Check whether monitoring detected each successful attack and time containment.
- Rate findings by impact through identity and fix with scopes, authorisation and approvals first.
- Run the scenario set as a regression suite on every model, prompt, tool or permission change.
Standards and references
- OWASP GenAI Security Project: OWASP Top 10 for Agentic Applications for 2026
- OWASP GenAI Security Project: LLM01:2025 Prompt Injection
- OWASP GenAI Security Project: LLM06:2025 Excessive Agency
- OWASP: Top 10 Non-Human Identities Risks 2025
- MITRE ATLAS: ATLAS data repository
- NIST: AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0)
- NIST: AI 600-1, Generative Artificial Intelligence Profile
- NIST: SP 800-53 Rev. 5, Security and Privacy Controls for Information Systems and Organizations
- NIST: The NIST Cybersecurity Framework (CSF) 2.0
- Model Context Protocol: Security Best Practices
- Cloud Security Alliance: Agentic AI Threat Modeling Framework: MAESTRO
- Microsoft Learn: Planning red teaming for large language models (LLMs) and their applications
Related NHI Mgmt Group resources: AI Red Teaming Guide · AI Agent Threat Modelling Guide · Multi-Agent and A2A Security Guide · AI Agent Memory Security Guide · AI Agent Tool and Plugin Security Guide