Subscribe to the Non-Human & AI Identity Journal

Notifications
Clear all

AI agent purple teaming: what breaks when identity state drifts?


(@nhi-mgmt-group)
Member Moderator
Joined: 1 year ago
Posts: 12212
Topic starter  

TL;DR: An AI agent executed a Scattered Spider style purple team exercise in AWS, created a new IAM user, attached administrator privileges, generated access keys, and triggered multiple detections within minutes, according to Permiso Security. The bigger lesson is that autonomous execution can outpace identity-state continuity, so access review and identity-switching assumptions need rethinking.

NHIMG editorial — based on content published by Permiso Security: Can an AI Agent Run a Purple Team Exercise? Hear Ye, Hear Ye

By the numbers:

Questions worth separating out

Q: How should security teams govern AI agents in purple team exercises?

A: Treat the agent as a governed identity, not a script.

Q: Why do AI agents complicate identity attribution in cloud environments?

A: Because an agent can create or use multiple identities inside one task while logs still show a single initiating session.

Q: What breaks when autonomous agents do not switch to the identity they created?

A: The exercise loses identity fidelity.

Practitioner guidance

  • Define identity handoff rules for autonomous test agents Require the agent to switch to the newly created identity when the task calls for it, and verify that downstream actions are executed only under that identity.
  • Correlate federated sessions with local IAM creation Join Okta federation logs, IAM user creation, access-key issuance, and policy attachment into one reviewable sequence.
  • Alert on privilege escalation followed by long-term key creation Treat administrator policy attachment plus new access-key generation as a compound signal, not two unrelated admin events.

What's in the full article

Permiso Security's full blog post covers the operational detail this post intentionally leaves for the source:

  • The step-by-step AWS attack chain Rufio executed, including IAM user creation, privilege attachment, and access-key generation.
  • The detection summaries and alert correlations Permiso used to reconstruct the full exercise timeline.
  • The daily state-diff method used to track how the agent evolved over twelve days.
  • The practical discussion of where AI agents help defenders and where they still struggle with context switching and identity state.

👉 Read Permiso Security's analysis of AI agent purple teaming and AWS identity state →

AI agent purple teaming: what breaks when identity state drifts?

Explore further

View Full Forum →  |  NHI Foundation Course →



   
Quote
(@mr-nhi)
Member Moderator
Joined: 2 months ago
Posts: 11787
 

Autonomous purple teaming does not just test detections, it tests whether identity governance can still track who is acting. When an agent can translate a threat narrative into live cloud actions, the control question shifts from "did we detect it" to "did we preserve identity continuity across the action chain." That is why autonomous test agents need the same governance seriousness as other operational identities. Practitioner conclusion: build identity-aware guardrails before experimentation expands into production-adjacent workflows.

A few things that frame the scale:

  • 88.5% of organisations acknowledge that their non-human IAM practices lag behind or are merely on par with their human identity and access management efforts, according to The 2024 Non-Human Identity Security Report.
  • 23.5% of security professionals are unsure about the biggest threat to their non-human identities, indicating a significant awareness gap.

A question worth separating out:

Q: Who is accountable when an autonomous agent generates privileged access in AWS?

A: Accountability sits with the team that scoped, approved, and monitored the agent, because the agent cannot own policy or governance decisions. In practice, cloud identity teams need audit trails that show who authorised the task, which identity executed each step, and where the privilege escalation occurred.

👉 Read our full editorial: AI agents can run purple team exercises, but identity state still breaks



   
ReplyQuote
Share: