AI red teaming exposes the security gaps in enterprise agents

At a glance

What this is: Obsidian Security says AI red teaming is needed because conventional penetration testing does not adequately expose prompt injection, model inversion, and agent workflow abuse in enterprise AI systems.

Why it matters: IAM, PAM, and AI security teams need a validation method that tests how AI agents access data, take actions, and persist across workflows, not just whether a static application responds securely.

Context

AI red teaming is specialised offensive testing for AI systems. It looks for weaknesses that appear only when models, prompts, tools, and workflows interact, rather than when a system is treated like a static web application. For identity and access teams, the important point is that enterprise agents can behave like privileged non-human identities, so access scope and action boundaries become security controls, not implementation details.

Traditional penetration testing still matters, but it was built for conventional application and infrastructure failure modes. AI deployments create a different governance gap because the risk is not only data exposure, but also manipulated reasoning, unsafe tool use, and unintended delegation across workflows. That makes validation of AI access, authorization, and monitoring part of the security design, especially where agents touch sensitive data or business processes.

The article’s starting position is typical of current enterprise AI deployments: organisations are adding AI capability faster than they are adapting testing and governance. That mismatch is where control assumptions break down.

Key questions

Q: What fails when AI red teaming is not in place for enterprise agents?

A: Without AI red teaming, enterprises often miss prompt injection, model inversion, and workflow manipulation that only appear when an agent is asked to reason and act across tools. The failure is not just weaker detection, but a false belief that static application testing is enough for dynamic, delegated systems. That leaves sensitive data access and business actions under-tested.

Q: Why do AI agent deployments create new risk even when the underlying models are well secured?

A: AI agent risk often comes from what the agent can reach and do, not only from model quality. If an agent has broad access to systems, data, or tools, it can expose sensitive information or perform unintended actions. Security teams need governance that covers identity, access, data boundaries, and auditability across the full runtime environment.

Q: What are the signs that AI agent security controls are too weak?

A: Common warning signs include agents accessing systems or data outside their intended scope, sharing sensitive information inappropriately, and using credentials or tools without clear justification. Another indicator is inconsistent visibility into what the agent touched, especially when security, compliance, and legal teams do not share the same view of agent activity and data access.

Q: How should organisations govern AI red teaming alongside AI security posture management?

A: Use red teaming to discover how an agent can be manipulated, then use posture management to keep inventory, permissions, and monitoring aligned with what was found. Together they create a control loop: test, remediate, observe, and retest. That approach is most effective when AI systems have production access to data or operational workflows.

Technical breakdown

Why traditional penetration testing misses AI agent failure modes

Conventional penetration testing assumes a system has relatively stable inputs, business logic, and output boundaries. AI agents break that assumption because prompts can reshape the control flow, tool selection, and even the interpretation of policy. A red team therefore has to test prompt injection, model inversion, and adversarial inputs as security conditions, not edge cases. The question is not only whether the model answers correctly, but whether it can be manipulated into taking prohibited actions, revealing hidden context, or moving outside the intended task boundary. Practical implication: test the AI workflow, not just the hosting stack.

Practical implication: validate agent behaviour across prompts, tools, and permissions, not only at the network or app layer.

How agent workflow manipulation creates security exposure

Agent workflow manipulation exploits the fact that an AI agent may chain reasoning, tool calls, and external actions without a human reviewing each step. Once an attacker influences that chain, the system can retrieve sensitive data, trigger downstream actions, or pass compromised context between steps. This is especially important when the agent has access to billing, customer, SaaS, or internal knowledge systems. In identity terms, the agent is acting like a non-human identity with delegated authority, so the attack surface includes authorisation scope, session context, and tool access. Practical implication: constrain what the agent can do per task and per tool.

Practical implication: scope each agent’s delegated access tightly and review every tool permission as if it were a privileged service account.

Why AI red teaming belongs in DevSecOps and MLOps

AI security testing becomes most useful when it is embedded into delivery pipelines rather than run as a one-off assessment. That means automated checks during model development and validation, plus deeper manual assessments before production release and after major workflow changes. The operational value comes from linking findings to remediation, risk reporting, and continuous monitoring so that teams can see when an agent’s behaviour drifts or a new integration expands its blast radius. For organisations using AI in regulated or customer-facing workflows, this is the point where testing becomes governance. Practical implication: make red teaming a release gate for high-risk AI capabilities.

Practical implication: add AI red team findings to release governance, incident monitoring, and change control for every material model or agent update.

Threat narrative

Attacker objective: The attacker aims to coerce an AI system into exposing sensitive information or carrying out actions outside its intended authority.

1. Entry occurs when an attacker supplies adversarial prompts, crafted inputs, or malformed instructions to an enterprise AI system.
2. Credential access or action abuse follows when the agent is manipulated into revealing sensitive context, using tools unsafely, or following hidden instructions.
3. Impact occurs when the agent leaks proprietary data, performs unauthorised actions, or propagates bad decisions across connected business workflows.

Breaches seen in the wild

• Langflow Flodrix botnet 2025: Attackers used Langflow's unauthenticated code endpoint to dump AI servers' environment variables and enlist them in the Flodrix botnet.
• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI red teaming is now an identity governance problem, not only a testing problem. Once an enterprise agent can access data, call tools, and act across systems, it behaves like a privileged non-human identity. That means validation has to examine authorisation scope, tool permissions, and action boundaries as part of governance, not as an afterthought. Practitioners should treat AI red teaming results as evidence about delegated authority, not just model quality.

Traditional testing fails because it assumes static behaviour. AI systems can change their output path, context usage, and tool choice based on the prompt environment, which means the risk surface is dynamic. That makes prompt injection, model inversion, and workflow manipulation distinct from ordinary application flaws. The field needs testing that follows the decision chain, not just the code path. Practitioners should align testing depth to the autonomy and data access of each agent.

Agent workflow manipulation is the most useful named concept in this article. It captures the point where an attacker uses the agent’s own orchestration logic against it, turning a productivity feature into a control bypass. The problem is not merely that an AI system can answer incorrectly, but that it can execute the wrong action with legitimate-looking authority. Practitioners should focus governance on what the agent can do after a prompt lands, not only on what it says.

Continuous AI validation will increasingly sit alongside DevSecOps and MLOps controls. As enterprises move AI into customer service, internal operations, and decision support, point-in-time assessments will not be enough. Security programmes will need repeatable red team cycles, monitoring, and change-triggered retesting to keep pace with model updates and new integrations. Practitioners should build AI testing into release and monitoring governance before the agent becomes business critical.

The category is converging on broader AI security posture management. Red teaming finds the failure modes, but organisations also need inventory, configuration control, and ongoing monitoring to prevent regression. That is where the identity angle matters most: every agent, connector, and token becomes part of the control surface. Practitioners should manage AI systems as governed services with explicit identity, access, and audit boundaries.

From our research library:

• Gartner predicts that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing rising costs, unclear value and insufficient risk controls.
• Read next: Agentic AI Security Guide

What this signals

Agent workflow manipulation: this is the control problem that emerges when a model can chain prompts, tools, and actions without a human checkpoint. The practical challenge is not whether the model is clever, but whether its authority is bounded tightly enough that a manipulated step cannot become a system-level incident. For teams running AI in production, governance has to move from output review to permission design, change control, and monitored execution.

AI red teaming will push security teams to treat agents as governed services with explicit identity and access boundaries. That matters because test findings are only durable when they feed back into provisioning, monitoring, and approval workflows, especially where an agent can reach sensitive data or SaaS actions. The 17% incident rate vs 76% for over-privileged systems, according to the 2026 Infrastructure Identity Survey, is a reminder that privilege scope still drives outcome even when the subject is an AI system.

For practitioners

• Map each AI agent to a delegated authority model Document what data, tools, and actions each agent can reach, then classify those permissions as bounded non-human access rather than generic application access.
• Red team prompts, tools, and workflow handoffs Test for prompt injection, model inversion, and agent workflow manipulation across the full interaction chain, including connectors and downstream automation.
• Gate production release on AI security findings Require red team results and remediation evidence before enabling customer-facing or high-impact agent capabilities in production.
• Monitor agent behaviour for privilege drift Track whether agents begin using broader context, new tools, or unexpected pathways after deployment, and treat drift as a governance event.

Key takeaways

• AI red teaming addresses failure modes that conventional penetration testing often misses, especially when prompts can alter tool use and workflow decisions.
• Enterprise agents should be governed like privileged non-human identities because their value comes from delegated action, not just model output.
• The strongest programmes combine red teaming, permission scoping, and continuous monitoring so that AI security does not stop at deployment.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Agent Workflow: An agent workflow is a machine-driven sequence where an AI system can call tools, access data, and continue execution with limited human involvement. These workflows need governance that sees API activity and tool use, not just browser sessions or user clicks.
• Model Inversion: Model inversion is an attack technique that tries to recover sensitive information from a model by querying it in a targeted way. The risk matters because information hidden in training data, prompts, or internal behaviour can sometimes be inferred from outputs rather than directly accessed.

Deepen your knowledge

The NHI Foundation Level course covers NHI governance, agentic AI identity, and machine identity security through the industry's only accredited NHI security programme. It is designed for security leaders who need to govern delegated access, not just model output.