AI red teaming is exposing gaps in model and agent governance

At a glance

What this is: This is a guide to AI red teaming that shows how adversarial testing exposes weaknesses in models, applications, and agents before production use.

Why it matters: It matters because IAM and security teams need to govern AI systems as dynamic decision-makers, not static software, or they will miss prompt, data, and guardrail failures.

Context

AI red teaming is a structured way to test how an AI system behaves under adversarial prompts, unsafe inputs, and boundary-pushing scenarios. In this article, the governance gap is that traditional cybersecurity tools do not fully address probabilistic model behaviour, prompt injection, or runtime drift in AI systems.

The identity question is not just whether the model is secure at build time. It is whether access, outputs, and policy enforcement still hold when the system is used in real workflows and its behaviour changes over time. That makes AI red teaming relevant to model governance, agent governance, and broader enterprise identity controls.

Key questions

Q: What should security teams do first after AI red teaming finds a weakness?

A: Start with the failures that can expose sensitive data, trigger unsafe actions, or undermine trust in a high-stakes workflow. Then separate model issues from application and agent issues so fixes land at the right layer. The goal is to reduce the most damaging exposure before expanding coverage.

Q: Why do agentic AI systems require continuous red teaming?

A: Agentic systems change when models are retrained, prompts are updated, retrieval sources shift, or new integrations are added. Those changes can reopen vulnerabilities even if the model was previously tested. Continuous red teaming is how teams detect drift, validate fixes, and keep security evidence aligned to the current system state.

Q: What breaks when AI guardrails are only tested before deployment?

A: Static testing misses production drift, new jailbreak variants, and data leakage patterns that appear only under real traffic. When controls are never validated live, false confidence grows while the model learns from contexts the test set never covered. Continuous monitoring closes that gap by turning live incidents into new test cases.

Q: How should teams govern AI agents without treating the model as the only control point?

A: Governance should follow the full runtime, not just the model checkpoint. Teams need approval gates for tool use, version control for prompts and procedures, isolation for sub-agents, and traceability for every governed run. If the harness changes, policy changes, even when the model does not.

Technical breakdown

Why AI red teaming is different from traditional red teaming

AI red teaming differs from classic adversarial testing because the target is not a fixed software path but a probabilistic system whose outputs depend on training data, prompts, and runtime context. The article distinguishes single-blind AI testing from double-blind traditional red teaming, where the goal is to expose how the model behaves under known testing conditions. That matters because the failure surface includes harmful completions, leaked data, and unsafe tool behaviour, not just code defects.

Practical implication: test the model, prompts, and guardrails together rather than treating AI security as a code-scanning problem.

How prompt injection and jailbreaking break guardrails

Prompt injection and jailbreaking work by steering the model away from intended policy boundaries. Injection alters the instruction set the model follows, while jailbreaking tries to bypass built-in safeguards so the system performs disallowed actions or generates unsafe content. In agentic systems, that risk grows because a manipulated prompt can influence tool use, data retrieval, or downstream action selection. The control failure is not only in the model response but in the trust chain around what the model is allowed to do next.

Practical implication: validate whether guardrails still hold when prompts attempt to override policy, tool scope, or content restrictions.

Why model drift creates a governance problem over time

The article notes that a model safe yesterday may become unpredictable tomorrow, especially when it keeps learning or adapting from user interaction. That is a governance problem, not just a testing problem, because assurance at one point in time does not guarantee safe behaviour after data, prompts, or use cases change. Continuous red teaming is therefore a monitoring discipline for model behaviour, not a one-time certification exercise.

Practical implication: treat red teaming as an ongoing control tied to lifecycle change, not a pre-launch checkbox.

Breaches seen in the wild

• 12,000 secrets in LLM training data: Truffle Security found 11,908 live API keys and passwords hard-coded in web pages captured by Common Crawl, a dataset used to train LLMs.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI red teaming is now a governance control, not just a security test: the article shows that model behaviour, prompt handling, and downstream agent actions all need adversarial validation before production use. Traditional testing assumes stable software behaviour, while AI systems can shift as prompts, data, and usage patterns change. The practical conclusion is that AI risk now sits inside the identity and governance lifecycle, not outside it.

Prompt injection is a control-boundary failure, not merely bad input handling: once a model can be steered into unsafe output or unintended tool use, the real issue is that policy enforcement has moved from explicit rules to probabilistic compliance. That makes trust in the model's output conditional and fragile. Practitioners should interpret prompt injection as evidence that guardrails are part of the security boundary, not an optional add-on.

Continuous red teaming exposes a runtime governance gap: the article makes clear that AI systems are not evaluated once and then frozen in place. Behaviour can drift as models adapt, users change, and deployments expand into higher-stakes use cases. That creates a persistent assurance gap that belongs in operational governance, lifecycle review, and change management.

Model behaviour creates identity risk when systems can act on behalf of users or businesses: the more an AI system influences decisions, retrieves data, or triggers actions, the more governance must cover who or what is authorised to do so. The article's emphasis on models, applications, and agents points to a broader control problem where identity, policy, and execution are converging. Practitioners need controls that evaluate runtime behaviour, not just initial access.

Agent governance will follow model governance, not the other way around: red teaming agents matters because agentic systems inherit model weaknesses and then amplify them through tool use and action chains. That means agent governance cannot be bolted on after deployment. The field is moving toward behavioural assurance across the full AI stack, and identity teams should expect their scope to expand accordingly.

What this signals

AI red teaming is becoming a lifecycle control: the main governance lesson is that AI assurance cannot stop at model approval. As deployments expand into decision-making and agentic workflows, practitioners need continuous validation tied to prompt changes, retraining, and new tool access.

The practical boundary is shifting from static policy review to runtime behaviour review. That means security teams should expect red teaming to sit alongside change management, access governance, and operational monitoring rather than being treated as a separate specialist exercise.

For practitioners

• Test for prompt injection across real workflows Exercise the model with adversarial prompts that try to override instructions, extract hidden context, or coerce disallowed actions in the same flows users will actually use.
• Validate jailbreaking resistance before production Check whether safety boundaries still hold when prompts are crafted to bypass refusals, content filters, or role restrictions, especially in high-stakes deployments.
• Prioritise data leakage findings first Rank red team results by the harm they could cause, with leaked secrets, training data, or user information treated as higher risk than cosmetic output failures.
• Repeat testing after model or workflow changes Re-run adversarial tests whenever the model is retrained, the prompt set changes, tool access expands, or a deployment moves into a new business process.

Key takeaways

• AI red teaming exposes failure modes that traditional software testing does not capture, especially around prompts, model behaviour, and agent actions.
• The article's core warning is that AI risk changes over time, so security validation must be continuous and tied to lifecycle change.
• For practitioners, the control question is no longer whether the model passed a test once, but whether it still behaves safely in real workflows.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Jailbreaking: Jailbreaking is the practice of crafting prompts that persuade an AI model to ignore its safeguards and produce restricted outputs. It shows that authentication to the service does not guarantee safe behavior, which is why governance must extend beyond the chat interface.
• Model Drift: Model drift is the gradual change in a model’s behaviour or performance after deployment. It happens when the operating environment, user patterns, or inputs no longer match the conditions used to validate the system. Drift matters because a model can appear functional while no longer meeting approved standards.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.