AI red teaming fails when testing stops at pre-launch

At a glance

What this is: This is an analysis of why pre-deployment AI red teaming is insufficient and why effective testing must continue across runtime and post-launch phases.

Why it matters: It matters because IAM, governance, and security teams need to account for AI systems that behave dynamically, interact with users, and call tools after release.

By the numbers:

• In February 2026, Cisco reported an average jailbreak success rate of 64% across models.
• Cisco found peak success rates for open-source models reached 93% during multi-turn attacks.
• A joint paper from researchers at OpenAI, Anthropic, and Google DeepMind found adaptive attacks succeeding more than 90% of the time against published model defenses.

👉 Read ActiveFence's analysis of why AI red teaming must extend beyond the lab

Context

AI red teaming is meant to expose failure modes before they reach production, but that promise breaks down when testing is limited to a single point in time. Once models are deployed, prompt patterns change, user behaviour shifts, model upgrades alter outputs, and AI agents begin interacting with tools and data sources that were never fully exercised in the lab.

The governance problem is not just model quality, but control over behaviour across the full lifecycle. For identity teams, that includes who can invoke the system, what the system can access, and how quickly policy, logging, and containment adapt when the AI begins acting outside the expected boundary. The article's starting position is increasingly typical, not exceptional, in enterprise AI programmes.

The primary keyword here is AI red teaming, but the deeper issue is runtime drift. That drift creates a control gap between pre-launch evaluation and the actual identity and access patterns the system exhibits in production.

Key questions

Q: How should security teams use continuous automated red teaming in practice?

A: Use it as a control verification loop, not as a substitute for human red teaming. The practical aim is to retest high-value detections, access paths, and policy changes whenever the environment changes, so security teams know whether a control still works after rollout rather than weeks later.

Q: Why do AI agents create a governance problem for IAM teams?

A: AI agents create a governance problem because they authenticate and act as autonomous software entities with tool access. If their actions are logged only as application activity, teams lose accountability, context, and revocation clarity. IAM must therefore extend to agent identity, delegated authority, and control-plane audit trails.

Q: What breaks when AI controls stop at pre-deployment testing?

A: Pre-deployment testing cannot stop a compliant model from making risky decisions in a live workflow or through connected tools. That leaves regulated data exposure, agent misuse, and weak audit trails unaddressed. Teams need controls that follow the interaction at runtime, because many failures only appear once the system is operating with real users, real data, and real permissions.

Q: Who is accountable when an AI system makes a harmful decision?

A: Accountability should follow the identity chain that authorized, configured, or triggered the action, including the human owner, the platform team, and any delegated agent or tool account. If the organisation cannot name that chain, the governance model is too weak for regulated AI use.

Technical breakdown

Why pre-launch AI red teaming misses post-release behaviour

Pre-launch red teaming validates a point-in-time version of the model against a finite set of adversarial scenarios. It can uncover jailbreaks, harmful outputs, and obvious policy bypasses, but it cannot fully simulate the changing conditions of real deployment. Model updates, prompt chaining, user adaptation, and external tool access all expand the attack surface after release. In practice, the failure is not lack of testing, but overconfidence in a test environment that cannot reproduce live operational drift.

Practical implication: treat pre-launch testing as one input to a lifecycle control program, not as proof of production safety.

How AI agents expand the attack surface beyond the model

An AI agent is not just a model that answers questions. It is a system that can choose actions, call APIs, and combine tools across workflows, which means the security boundary extends into permissions, integrations, and downstream services. Once agents are involved, behavioural abuse can become access abuse, especially when identity, authorization, and logging are not designed for autonomous or semi-autonomous action. This is where AI security starts to intersect with IAM and NHI governance.

Practical implication: inventory every tool, credential, and approval path an AI agent can touch, then govern them as identities with scoped access.

Why runtime monitoring and post-launch enforcement matter

Runtime controls exist because discovery at design time is always incomplete. Guardrails, monitoring, and post-launch adversarial testing create a feedback loop that catches drift, regressions, and newly discovered attack patterns after the system is live. Without that loop, teams learn about failures only after users, attackers, or model updates expose them. The operational issue is not just detection speed, but whether enforcement can happen fast enough to limit blast radius.

Practical implication: pair continuous evaluation with enforcement actions that can disable, constrain, or revoke AI capabilities in production.

Threat narrative

Attacker objective: The attacker seeks to make the AI system behave in ways that bypass policy, produce harmful outputs, or misuse connected tools and data.

1. Entry occurs when adversaries exploit behavioural weaknesses in deployed AI systems after pre-launch testing has already been completed.
2. Escalation happens when the model drifts, agents gain broader tool access, or attackers chain prompts and workflows that testers did not model.
3. Impact follows when harmful outputs, policy violations, or unauthorized actions affect users, data, or downstream business systems.

NHI Mgmt Group analysis

AI red teaming is becoming a lifecycle governance problem, not a one-time evaluation exercise. The article is right to distinguish tests from continuous security assurance. In AI programmes, the gap between pre-launch testing and production drift is where policy failures, unsafe outputs, and unauthorised tool use emerge. Practitioners should treat this as an operating model issue, not a point solution problem.

Agentic AI identity is the governance boundary most teams still under-estimate. Once an AI system can call tools, chain actions, or act on behalf of users, it begins to resemble a non-human actor with delegated access. That creates a direct intersection with NHI governance because the system's permissions, tokens, and approval paths become as important as its model behaviour. Practitioners should classify AI agents as identity-bearing systems with scoped authority.

Runtime assurance is the missing control layer in most AI programmes. The article's strongest point is that detection only matters if enforcement can follow quickly enough to reduce blast radius. NIST AI RMF planning, measurement, and management functions all point toward ongoing monitoring rather than static validation. Practitioners should build controls that can constrain or revoke risky AI behaviours after deployment.

Continuous red teaming should be measured by learning velocity, not test volume. The real question is how fast a team identifies, understands, and closes a failure once production exposes it. That shifts the programme from periodic benchmarking to operational resilience, with clear ownership across security, engineering, and product teams. Practitioners should measure how quickly a live AI control can be tightened when behaviour changes.

Security and safety cannot remain separate disciplines once AI systems interact with users and tools. The article shows that harmful content, policy violation, and access abuse now sit inside the same operational chain. That convergence matters for governance because separate review tracks create blind spots and duplicated assumptions. Practitioners should align AI safety review with IAM, logging, and incident response so the control model matches the system's actual behaviour.

What this signals

Runtime drift is now the practical boundary for AI governance. Organisations that only test pre-production models will keep missing the control gap that opens once prompts, tools, and users change in live environments. The programme implication is clear: continuous evaluation, constrained delegation, and fast rollback need to become part of the operating model, not an emergency response.

Agentic AI will force identity teams to govern systems, not just people. When an AI system can act, the questions become who delegated authority, what it can touch, and how that authority is revoked when behaviour changes. That makes NHI-style lifecycle thinking relevant to AI governance, especially where service accounts, APIs, and tokens underpin automation.

Model risk and access risk are converging into one operational problem. The more an AI system can interact with data and tools, the more security teams need shared telemetry across IAM, logging, and response workflows. Practitioners should prepare for control models that combine AI testing, identity governance, and incident containment in a single loop.

For practitioners

• Build continuous AI testing into the production lifecycle Run adversarial checks before launch, at runtime, and after model or prompt changes so drift is tested as a standing condition, not a one-off review.
• Map every agent tool and credential path Document which APIs, systems, and service tokens an AI agent can reach, then apply least privilege and separate approval for higher-risk actions.
• Create enforcement controls that can fail closed Design guardrails so policy violations can block responses, disable tools, or restrict delegation before harmful actions propagate downstream.
• Measure learning velocity as a programme metric Track the time between a discovered failure and the corresponding policy, prompt, or control update, because that interval reveals true resilience.
• Include security, product, and safety owners in one review loop Assign shared accountability for AI behaviour, logging, and incident response so model testing is not separated from operational ownership.

Key takeaways

• AI red teaming that ends at launch misses the production conditions where most failures emerge.
• The evidence from current testing shows that static defences and one-day reviews are not keeping pace with adaptive attacks.
• Practitioners need continuous evaluation, runtime enforcement, and identity-aware governance for agents and their tool access.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Runtime Drift: Runtime drift is the gap between an AI agent’s approved authority and its actual behaviour as conditions change. It appears when the agent adapts to new context, new integrations, or new instructions and begins acting outside the scope that governance originally defined.
• Agentic AI Identity: The complete set of credentials, permissions, and governance controls applied to an autonomous AI agent — covering authentication, authorisation, action logging, and access revocation. Distinct from traditional NHI because agent identities are often ephemeral, delegated, and multi-hop.
• Learning Velocity: Learning velocity is the speed at which an organisation detects, understands, and responds to a newly observed failure in production. In AI security, it is a better indicator of resilience than test frequency alone because it measures whether the programme can adapt as quickly as the threat landscape changes.

What's in the full article

ActiveFence's full blog covers the operational detail this post intentionally leaves for the source:

• Mo Sadek's live talk framing on why enterprise red teaming fails when it is not continuous
• The full pre-launch, runtime, and post-launch control model used to structure the argument
• The five practical directives on ownership, drift, and learning velocity in their original context
• The related AI red teaming resources and product-specific walkthroughs referenced at the end of the post

👉 ActiveFence's full post covers the runtime control model, drift problem, and practical testing guidance.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and secrets management. It helps security practitioners connect identity controls to the systems and workflows that now depend on delegated access.