AI red teaming exposes the security gaps in enterprise agents

At a glance

What this is: This is an Obsidian Security analysis of why AI red teaming is becoming necessary as enterprises deploy more LLMs, autonomous agents, and machine learning pipelines.

Why it matters: It matters because AI systems introduce new attack paths, including prompt injection and workflow abuse, that intersect with IAM, secrets, and governance when agents access sensitive data or perform actions.

By the numbers:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, and as quickly as 9 minutes in some cases.

👉 Read Obsidian Security's analysis of AI red teaming services for enterprise AI resilience

Context

AI red teaming is the specialised practice of attacking AI systems to find weaknesses before adversaries do. The gap it addresses is straightforward: conventional penetration testing assumes stable application behaviour, while AI systems can alter outputs, actions, and data access in response to crafted prompts or poisoned inputs.

For identity and access teams, the critical issue is not only model safety but also the identity and privilege model around the model itself. When an agent can reach data, tools, or downstream systems, AI security becomes an access-governance problem as much as a testing problem. In enterprise environments, that intersection is now common rather than exceptional.

The article’s starting position is typical for current enterprise AI deployments: organisations are adopting AI faster than they have built repeatable testing and governance controls.

Key questions

Q: How should security teams test generative AI systems for real-world abuse?

A: They should test the full execution path, not just the model prompt. That means probing retrieval, tool use, output handling, and any delegated actions the system can perform. The goal is to reproduce how an attacker would move from manipulation to impact, then capture whether the system leaks data, ignores guardrails, or performs unauthorised actions.

Q: Why do AI agents complicate access governance more than ordinary automation?

A: AI agents complicate access governance because they can branch at runtime, wait on external services, and continue later with the same operational context. That means privilege is not just granted at launch, it persists across a live session that must be observable, resumable, and attributable.

Q: What breaks when AI red teaming is not connected to DevSecOps?

A: Testing becomes a point-in-time exercise with no lasting control effect. New prompts, connectors, data sources, or model updates can introduce fresh attack paths after the assessment is finished. Without pipeline integration, security teams may discover issues in reports but never force changes to deployment gates, monitoring rules, or privilege settings.

Q: Who is accountable when an authorised AI agent causes a breach?

A: Accountability usually sits with the organisation that assigned the access, defined the workflow, and failed to instrument runtime oversight. The hard part is proving whether the failure was an entitlement decision, a workflow design issue, or a missing behavioural control, which is why governance ownership must span IAM, security engineering, and application teams.

Technical breakdown

Why traditional penetration testing misses AI attack paths

Traditional penetration testing is designed to validate infrastructure, application logic, and known exploit classes. AI systems add behaviours that are probabilistic and instruction-sensitive, which means a model can be manipulated without exploiting a conventional software flaw. Prompt injection can steer model output and actions, while model inversion and adversarial inputs target the model’s learned behaviour rather than its code. That is why AI red teaming uses attack simulation against the model, the prompt chain, the tools it can call, and the data it can reach. The goal is to test whether the system remains safe when its reasoning context is intentionally corrupted.

Practical implication: validate AI systems with AI-specific attack scenarios, not just standard application scans.

How autonomous agent workflows expand the attack surface

Autonomous and semi-autonomous agents widen the security boundary because they can make decisions, invoke tools, and persist across interactions. That creates multiple control points: prompt input, tool selection, identity and access boundaries, and memory or context storage. If those controls are weak, an attacker does not need full system compromise. Instead, they can influence the agent to reveal data, execute unintended actions, or escalate into connected services. In identity terms, the agent’s runtime privileges become the real security perimeter, especially where the agent inherits secrets, OAuth tokens, or broad API permissions.

Practical implication: treat agent runtime privileges and tool access as first-class security controls.

Why AI red teaming must connect to DevSecOps and MLOps

AI red teaming has limited value if findings sit outside the delivery pipeline. Effective programmes connect testing to CI/CD and MLOps so that model changes, prompt updates, connector changes, and policy changes are re-evaluated before release. This matters because AI risk is not static. A model may be safe in one configuration and vulnerable after a new data source, plugin, or workflow is added. Continuous monitoring closes that gap by checking whether deployed systems drift away from the tested security baseline and whether new behaviours introduce fresh attack paths.

Practical implication: embed AI security testing into release gates and post-deployment monitoring.

Threat narrative

Attacker objective: The attacker wants to turn the AI system’s own reasoning and tool access into a data-exfiltration or workflow-abuse channel.

1. Entry occurs through crafted prompts, malicious inputs, or compromised integrations that reach the AI system’s instruction path.
2. Escalation happens when the model is persuaded to reveal sensitive data, misuse tools, or follow attacker-shaped workflows across connected services.
3. Impact follows when the agent exposes information, performs unauthorised actions, or corrupts business decisions at production scale.

NHI Mgmt Group analysis

AI red teaming is becoming an identity-governance discipline, not just a testing service. The article correctly frames prompt injection and model inversion as AI-specific attack types, but the deeper issue is who or what the system is allowed to act as. Once an agent can call tools, touch data, and persist context, security teams have to govern its runtime identity and permissions with the same seriousness they apply to privileged human access. Practitioners should fold AI red teaming into access governance, not keep it isolated in security testing.

Agent workflow abuse creates a new form of privilege drift. The article’s emphasis on autonomous workflows points to a control gap that most IAM programmes still treat as exceptional: permissions are granted to the agent once, then reused across changing tasks and contexts. That is a governance failure, because the effective privilege boundary moves faster than review cycles. Practitioners should expect agent privileges to drift unless they are bound to task scope, connector scope, and monitored execution context.

AI red teaming only matters if it produces enforceable controls. Testing that identifies prompt injection or data leakage is useful, but the value disappears when findings do not change policy, connector permissions, or release gates. The article’s DevSecOps emphasis is the right direction, yet the governance layer has to translate red-team outcomes into limits on data access, tool invocation, and model-to-system trust. Practitioners should make remediation as measurable as the test itself.

AI security posture now depends on the identity of connected systems. The article notes SaaS integrations, token compromise, and excessive privileges, which means AI security cannot be separated from secrets management and non-human identity governance. When an agent is granted broad access through tokens or connectors, the model becomes only as safe as the underlying credential discipline. Practitioners should align AI red teaming with NHI controls and secret lifecycle management.

Prompt injection is the visible symptom, but uncontrolled delegation is the underlying concept. The real risk is not just that a model can be tricked, but that enterprise systems increasingly trust the model to decide when and how to use privileges. That creates an uncontrolled delegation layer between the user, the agent, and the downstream service. Practitioners should govern that delegation layer explicitly before scaling agent deployments.

What this signals

AI agent governance will increasingly be measured by what the agent can reach, not what the model can generate. The practical programme question is whether access boundaries, token hygiene, and connector approvals are enforced as living controls. Once agents start acting as non-human identities, identity teams need visibility into tool use, data access, and exception handling as part of the security baseline.

Agentic AI introduces a governance gap that looks familiar to IAM teams. It is a privilege problem disguised as a model problem. The closer an agent gets to production business workflows, the more the programme has to manage delegated access, short-lived credentials, and monitoring of runtime behaviour. That is where identity governance and AI security converge.

40% of your agents are a critical risk is the wrong headline to optimise against if the underlying trust model is unchanged. The real issue is whether the organisation can prove which agents have access, what they touched, and whether those permissions still make sense. Use the NIST AI Risk Management Framework to anchor governance ownership and decision accountability.

For practitioners

• Test agent workflows, not just model outputs Red-team the full path from prompt to tool call to downstream action. Include connector abuse, memory persistence, and data retrieval scenarios so the assessment reflects the agent’s real runtime behaviour.
• Bind privileges to task scope Limit each agent to the smallest set of tools, data sources, and APIs needed for a specific workflow. Review whether token inheritance or long-lived OAuth grants are creating standing access that outlives the task.
• Feed red-team findings into release gates Require AI security test results before model, prompt, connector, or policy changes can ship. Treat a failed red-team scenario as a block on production deployment until the control gap is closed.
• Track identity-linked exposure paths Map which AI systems can reach sensitive data, credentials, or privileged APIs, then tie that inventory to secrets rotation and access review cycles. Use the same governance discipline for agents that you use for other non-human identities.

Key takeaways

• AI red teaming is necessary because AI systems fail in ways traditional penetration tests are not designed to detect.
• The main security issue is delegated runtime privilege, especially when agents can call tools, reuse tokens, and reach sensitive data.
• Enterprises need AI testing, identity governance, and release controls to work together or the attack surface will keep expanding.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Agent Workflow: An agent workflow is a machine-driven sequence where an AI system can call tools, access data, and continue execution with limited human involvement. These workflows need governance that sees API activity and tool use, not just browser sessions or user clicks.
• Delegated Access: Delegated access is permission granted to one identity to act on behalf of another user, service, or system. In NHI environments, this usually appears in OAuth-connected apps and automation tooling. It is powerful, but it must be tightly scoped and reviewed because it can persist long after the original business need ends.

What's in the full article

Obsidian Security's full blog post covers the operational detail this post intentionally leaves for the source:

• Detailed breakdown of AI red teaming techniques for prompt injection, model inversion, API fuzzing, and agent workflow manipulation
• Examples of how to embed AI security testing into CI/CD and MLOps release gates
• Benchmark-style metrics for measuring vulnerability discovery, false positives, and testing coverage across agent workflows
• Discussion of how AI security testing connects to broader security posture management and remediation workflows

👉 Obsidian Security's full post covers testing methods, integration patterns, and measurement approaches for AI security programmes.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps security and identity practitioners build the control model that underpins safe automation.