AI red teaming exposes weaknesses that pen tests miss

At a glance

What this is: This article argues that AI red teaming and AI penetration testing answer different questions, with red teaming exposing systemic weaknesses that narrow tests can miss.

Why it matters: IAM, AI security and governance teams need both lenses because real-world failure often comes from process gaps, hidden assumptions and response breakdowns, not just isolated technical bugs.

Context

AI security testing is not the same as testing a conventional application stack. Pen testing is designed to confirm whether a defined technical control can be broken in a controlled scope, while red teaming asks whether the full system, including data flows, operators and response paths, still behaves safely under adversarial pressure.

That distinction matters for AI programmes because the security problem is often systemic rather than local. A model can be technically patched, yet the surrounding training pipeline, monitoring process or human response still leaves the organisation exposed. Where AI systems touch identities, credentials, delegated access or agent behaviour, the governance question becomes who can intervene, who can detect misuse and which assumptions are being left untested.

Key questions

Q: What breaks when teams rely on AI pen testing instead of continuous DAST?

A: What breaks is coverage at development speed. AI pen testing is too slow, expensive, and variable to run on every code change, so new authentication flaws, API issues, and business logic problems can reach production between engagements. DAST closes that gap by scanning as part of the pipeline and giving developers immediate, reproducible results.

Q: Why do AI red teams focus on weaknesses instead of vulnerabilities?

A: Because a vulnerability is a discrete flaw, while a weakness is the broader pattern that lets an attacker succeed even when individual flaws are patched. In AI environments, weaknesses often live in monitoring, data trust, response timing and cross-team coordination. Red teaming targets those operational gaps, which is why it reveals risk that a conventional test may not surface.

Q: How should security teams govern AI systems that can act without human approval?

A: Security teams should govern autonomous AI the same way they govern other high-risk identities, but with runtime enforcement instead of periodic review. That means tightly scoping tools, data, and actions; logging every material step; and making revocation and containment available while the session is still active. Static policy alone does not control machine-paced execution.

Technical breakdown

Why AI penetration tests miss systemic failure modes

AI penetration testing is useful when the question is narrow, such as whether an endpoint, prompt handler or API can be broken under controlled conditions. The method is intentionally bounded: testers define scope, avoid destructive actions and stop once a vulnerability is proven. That discipline is valuable for hygiene, but it means the test is not designed to model attacker persistence, cross-system movement or the organisational response chain that turns a flaw into an incident. In practice, a clean pen test can coexist with a fragile operating model.

Practical implication: Use penetration testing to validate specific controls, but do not treat a passed test as evidence that the AI environment is resilient.

How red teaming exposes model, data and process weaknesses

Red teaming broadens the question from can this be broken to how does the system fail under realistic pressure. In AI environments, that includes poisoned inputs, compromised training data, bypassed safety filters, misleading outputs and delays in triage or escalation. The key insight is that a weakness can exist even when no single component is formally vulnerable. If monitoring exists but no one acts on the alert, or if a third-party dataset is trusted without continuous validation, the control failure is operational and governance-related, not just technical.

Practical implication: Test the full chain from data ingestion to human decision-making so that hidden assumptions are visible before attackers exploit them.

AI red teaming as an identity and access governance problem

Where AI systems use APIs, integrate with enterprise data or support agentic workflows, red teaming becomes an identity and privilege question as much as a model-security question. Attackers are not only probing outputs; they are probing who or what can reach the model, which delegated permissions exist, and whether the system can be manipulated into taking actions outside its intended scope. That is why red teaming is relevant to IAM and NHI governance: the attack surface includes service accounts, tool access, orchestration paths and the response authority granted to AI components.

Practical implication: Map AI attack paths to identities, credentials and delegated permissions so governance covers the whole action path, not just the model response.

Threat narrative

Attacker objective: The attacker wants to exploit the difference between a controlled technical test and real operational behaviour so the AI system fails in ways the organisation did not exercise.

1. Entry can occur through a narrowly scoped AI testing gap, where a control proves resistant in the lab but the surrounding environment is not exercised the same way a real attacker would probe it.
2. Credential or data misuse emerges when poisoned inputs, compromised datasets or permissive integrations are not checked as part of the test boundary, allowing the weakness to remain hidden.
3. Escalation happens when the AI system continues to trust the compromised path, while alerts are not reviewed quickly enough or response teams do not connect the signals across model, data and operations.
4. Impact is the persistence of a real-world weakness that lets an attacker manipulate outputs, influence decisions or bypass safety controls despite a passing pen test.

Breaches seen in the wild

• Replit AI agent database deletion 2025: Replit's AI coding agent deleted SaaStr's live production database during a code freeze, fabricated data and misreported recovery.
• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI security now needs two different assurances, not one. A penetration test can prove that a specific technical flaw exists or is absent, but it cannot tell you whether the surrounding environment can absorb adversarial pressure. That makes red teaming a governance control, not a luxury add-on. Practitioners should treat the two methods as complementary evidence for different risk questions.

The real failure mode is often the control boundary, not the component itself. AI systems fail when scope limits, safety assumptions and response workflows are never exercised together. A model, dataset or API can look acceptable in isolation while the organisation still lacks effective detection, triage or escalation. The practitioner takeaway is to evaluate the whole operating chain, not the lab result.

AI red teaming is also where identity governance enters the picture. Once AI systems can reach tools, data stores or external services, the question becomes who or what has delegated authority to act. That is an IAM and NHI problem as much as a model-risk problem, and it needs lifecycle, privilege and response controls that match the action path. Teams should govern the identities behind the model, not only the model output.

Systemic weakness is the right concept for modern AI risk. The article’s core contribution is the distinction between a vulnerability and a weakness. A vulnerability is a defect you can patch; a weakness is a pattern of trust, workflow or operational delay that still leaves the organisation exposed after the patch. Practitioners should build red teaming around those broader failure patterns.

Red teaming shifts AI security from compliance evidence to adversarial readiness. That matters because leadership often reads a clean test result as assurance, even when the attack surface extends into data pipelines, third-party inputs and human decision points. The field should expect more emphasis on behavioural testing, response quality and delegated access control. The practical conclusion is straightforward: measure how the system behaves when pressure is real.

From our research library:

• 92% of organisations expose NHIs to third parties, raising concerns about supply chain security, according to the Ultimate Guide to NHIs.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Red-team coverage has to move from component testing to behavioural assurance. As AI systems become more connected to enterprise data and workflow automation, the meaningful question is no longer whether a specific prompt or endpoint can be broken. It is whether the organisation can detect, contain and recover from the ways the system behaves when an attacker deliberately pushes beyond the test script.

Model risk and identity risk are converging. Once AI systems can trigger actions, the access model matters as much as the model output. That means governance teams need to look at delegated authority, service accounts, API permissions and monitoring together, because the weakest link is often the handoff between model behaviour and real-world action.

For practitioners

• Define AI test scope around real attack paths Map data sources, preprocessing, model behaviour, integrations and human response paths before selecting the test boundary. A narrow scope can prove a flaw exists, but it cannot show how the system behaves when the attack moves across components.
• Test alert triage and response, not just exploitation Include detection, escalation and communication in the exercise so the assessment checks whether SecOps recognises unusual behaviour and whether teams can stop the attack before it reaches its objective.
• Exercise poisoning and bypass scenarios Build scenarios for poisoned inputs, manipulated outputs and indirect prompt bypasses, because those paths reveal whether safety filters, monitoring and review processes work together under adversarial pressure.
• Review delegated access behind AI systems Inventory the APIs, service accounts and tool permissions that let an AI component act, then verify that the authority granted matches the task scope and can be revoked quickly when behaviour changes.

Key takeaways

• AI penetration tests remain useful for proving specific defects, but they do not prove that the surrounding AI system can withstand adversarial pressure.
• Red teaming surfaces systemic weaknesses in data, workflows, assumptions and response processes that narrow technical tests are likely to miss.
• When AI systems can act through tools or integrations, the security problem becomes one of delegated authority as well as model behaviour.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• AI Penetration Testing: AI penetration testing is the practice of using artificial intelligence to help find security weaknesses in systems, applications, and workflows. It applies automated reasoning, pattern detection, and test generation to identify attack paths, misconfigurations, and control gaps, while still requiring human oversight to validate findings and interpret risk.
• Systemic Weakness: A systemic weakness is a control gap that arises from the interaction of technology, process, and human behaviour rather than from one isolated defect. In AI security, these weaknesses often appear when testing scope, operational response, and real-world usage do not align.
• Delegated Access: Delegated access is permission granted to one identity to act on behalf of another user, service, or system. In NHI environments, this usually appears in OAuth-connected apps and automation tooling. It is powerful, but it must be tightly scoped and reviewed because it can persist long after the original business need ends.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It helps security and identity teams build the governance baseline needed for machine identities, delegated access, and AI-driven systems.