AI red teaming exposes the security gaps in enterprise AI systems

At a glance

What this is: AI red teaming is a specialised testing approach for AI systems that targets model, agent, and data-flow weaknesses conventional penetration testing often misses.

Why it matters: It matters to IAM and security practitioners because AI agents, models, and connected workflows can expose sensitive data, invoke tools, and bypass access assumptions that were never designed for machine-driven decisioning.

By the numbers:

• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate.
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, and as quickly as 9 minutes in some cases.

👉 Read Obsidian Security's analysis of AI red teaming for enterprise AI systems

Context

AI red teaming is the structured practice of trying to break AI systems before attackers do. The reason it exists is simple: AI systems create new failure modes around prompt injection, model extraction, memory poisoning, and agent misuse that standard vulnerability testing does not reliably surface. For identity and access teams, the key issue is that AI systems increasingly act as decisioning and execution layers with their own access paths.

The security gap is not just technical, it is governance-related. When AI agents can read data, call APIs, and chain actions across tools, organisations need controls for identity, authorisation, observability, and remediation that extend beyond traditional application testing. That makes AI red teaming relevant to IAM, PAM, NHI governance, and AI security programmes at the same time.

This is a typical enterprise problem for teams moving from isolated AI experiments to production AI workflows. The article focuses on how testing should evolve as AI systems become part of operational processes, not just lab models.

Key questions

Q: How should security teams handle prompt injection in AI systems?

A: Treat prompt injection as an authorisation problem, not only a content problem. Validate user input, monitor for override patterns, and block any prompt that tries to change model policy, exfiltrate data, or trigger sensitive actions. The safest design is one where untrusted prompts cannot directly reach code execution, production data, or privileged tools without policy checks.

Q: Why do AI agents increase IAM and PAM risk?

A: AI agents increase IAM and PAM risk because they can execute actions quickly once privilege is available, which shortens the time available to detect misuse. If access is always on, the attack surface is always on too. That is why task-scoped privilege and ownership controls matter.

Q: What do organisations get wrong when they treat AI red teaming as a one-time assessment?

A: They assume the result stays valid after the model, prompts, data connectors, or orchestration logic changes. In practice, AI systems evolve quickly, so testing needs to be tied to deployment cycles and change events. Without that, the assurance outcome goes stale and gaps reappear between releases.

Q: How should teams account for AI red teaming under governance and compliance obligations?

A: Assign clear ownership, keep audit evidence for test scope and remediation, and connect findings to risk and change management records. AI security testing should support governance decisions, not sit outside them. That gives compliance teams traceability while helping engineering teams fix the specific control gaps that red teaming exposed.

Technical breakdown

How AI red teaming differs from conventional penetration testing

Traditional penetration testing assumes a mostly deterministic application stack with known inputs, known assets, and known vulnerability classes. AI red teaming instead probes probabilistic systems that can change output based on prompt structure, context window state, training data, and tool access. That is why adversarial input generation, model extraction attempts, and context manipulation matter. The tester is not only asking whether the service is reachable, but whether the model can be induced to reveal data, take unsafe actions, or drift from intended behaviour. For agentic systems, the risk extends to delegated tool use and chained execution.

Practical implication: test AI systems with adversarial scenarios that target decision-making, context handling, and delegated tool access, not only infrastructure flaws.

Why AI agents and connected workflows expand the attack surface

AI agents are not just interfaces to models. They can retrieve data, invoke tools, and move between systems, which creates a larger trust boundary than most application owners expect. Prompt injection can steer an agent into disclosing data or performing unauthorised actions. Memory poisoning can persist malicious context across sessions. Model inversion can expose training artefacts or sensitive inputs. Once the agent is connected to internal databases or external APIs, the security question becomes one of identity, privilege, and controlled execution, not just content moderation or model accuracy.

Practical implication: inventory every agent action path and treat each external tool connection as an access control decision.

How red teaming fits into MLOps and governance workflows

AI red teaming is most effective when it runs continuously, not as a one-off assurance exercise. In mature programmes, testing is embedded into MLOps so that model updates, retraining events, and deployment changes trigger fresh adversarial checks. Findings then flow into vulnerability management, risk registers, and audit evidence. This is important because AI systems evolve quickly, and control gaps can appear between training, deployment, and operational use. The governance challenge is to preserve traceability across the model lifecycle while still enabling rapid development.

Practical implication: connect red team outputs to MLOps gates, remediation ownership, and audit trails before AI systems move into production.

Threat narrative

Attacker objective: The attacker wants to make the AI system disclose sensitive information, perform unsafe actions, or execute workflows outside authorised intent.

1. Entry occurs through adversarial prompts, malformed API inputs, or manipulated context that reaches the AI system through a trusted interface.
2. Escalation follows when the model or agent is induced to reveal data, call restricted tools, or reuse context in ways that exceed intended privilege.
3. Impact comes from unauthorised data exposure, unsafe automated actions, or corrupted decisioning across downstream business workflows.

NHI Mgmt Group analysis

AI red teaming is becoming a control discipline, not just a testing exercise. Once AI systems can read data, call tools, and generate actions, the security question is whether those behaviours remain within governed bounds. That pushes red teaming into the same decision space as IAM, PAM, and NHI governance. The practical conclusion is that AI assurance now needs measurable controls, not just occasional adversarial testing.

Agentic workflows create a verification gap that traditional application testing cannot close. Prompt injection and context manipulation matter because they exploit the assumptions behind trust in tool-using systems. The named concept here is agent delegation trust gap: the mismatch between what an AI system is allowed to do and what the surrounding controls can actually observe. Practitioners should treat this as an authorisation and monitoring problem, not only a model safety issue.

Governance has to follow the AI lifecycle or it will miss the real risk window. AI systems are updated frequently, retrained often, and connected to live data and APIs after deployment. That means one-time validation quickly becomes stale. The field should move toward lifecycle controls that align testing, change management, and incident response across model development and production use.

Identity is now part of AI security whether teams planned for it or not. When an AI agent accesses internal databases or external services, its permissions, tokens, and service identities become part of the attack surface. This is where NHIMG's perspective matters most: the security model must distinguish between the model, the agent, and the identities the agent uses. Practitioners should govern agent identity as a first-class control object.

Continuous red teaming is a resilience measure, not a compliance checkbox. The article's core lesson is that security testing only works when it keeps pace with model change. Organisations that measure coverage, time to remediation, and workflow exposure are better positioned than those that treat AI assurance as a periodic review. The practical conclusion is to tie red teaming to operational resilience metrics and ownership.

What this signals

AI red teaming is likely to become a routine control in programmes that move AI agents into production. The practical challenge is not whether to test, but how to keep testing aligned to model updates, new tools, and changing data access paths. Teams that treat AI assurance as a change-managed control will see fewer blind spots than teams that rely on periodic review.

Agent delegation trust gap: the real governance problem is that many controls assume a clear separation between model output and system action. Once an agent can execute work, the permission model becomes part of the threat model. Security teams should align their testing and access controls with OWASP Top 10 for Agentic Applications 2026 and the NIST AI Risk Management Framework.

As agent use expands, the organisations that can show coverage, ownership, and remediation timelines will be the ones that can defend AI adoption to auditors and executives. A mature programme will look less like a model testing project and more like an identity and control governance discipline.

For practitioners

• Map AI agent identities and permissions Create a full inventory of every AI agent, service account, token, and API scope used in production workflows. Classify which identities can read data, invoke tools, or trigger downstream actions, then assign ownership for each credential and privilege set.
• Embed red teaming into MLOps gates Run adversarial tests whenever models are retrained, prompts change materially, or new tool integrations are added. Block promotion when prompt injection, model extraction, or unsafe tool use is reproducible.
• Measure workflow exposure, not just model accuracy Track which interaction paths are tested, which data sources are reachable, and which actions an agent can execute under stress. Use coverage and remediation metrics to show whether the agent can be manipulated into exceeding intended scope.
• Separate model risk from delegated access risk Treat model behaviour, agent orchestration, and attached credentials as distinct risk domains. An AI model can be safe in isolation while the agent built around it can still misuse data, tokens, or external tools.

Key takeaways

• AI red teaming matters because AI systems fail in ways conventional penetration testing does not model.
• The scale of the problem is growing as agent deployment accelerates and governance visibility remains uneven.
• The practical answer is continuous testing tied to MLOps, identity controls, and remediation ownership.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Model Inversion: Model inversion is an attack technique that tries to recover sensitive information from a model by querying it in a targeted way. The risk matters because information hidden in training data, prompts, or internal behaviour can sometimes be inferred from outputs rather than directly accessed.
• Agent Delegation Chain: A delegation chain is the sequence of agents that pass tasks, context, or actions to one another before work is completed. In governed systems, each hop becomes a security boundary because identity, authority, and accountability can change as the task moves downstream.

What's in the full article

Obsidian Security's full article covers the operational detail this post intentionally leaves for the source:

• Step-by-step AI red teaming techniques for prompt injection, model inversion, and adversarial input testing.
• Practical guidance on embedding security checks into MLOps and CI/CD workflows without disrupting release cadence.
• Metrics and benchmarks for measuring vulnerability discovery rate, remediation speed, and test coverage across AI systems.
• Platform integration detail for teams that want to correlate AI findings with broader security and governance workflows.

👉 Obsidian Security's full article covers the testing methods, governance integration, and benchmark metrics in more depth.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It is designed for practitioners who need to connect identity controls to modern AI and automation risks.