AI red teaming is exposing the limits of traditional security

At a glance

What this is: This article explains why AI red teaming is becoming a core control for exposing semantic attacks, prompt injection, data leakage, and agent misuse in modern AI systems.

Why it matters: It matters because IAM and security teams now have to govern AI behaviour, tool use, and data access across models, agents, and connected systems, not just traditional infrastructure.

By the numbers:

• 97% of organizations surveyed had experienced security issues tied to Gen AI in the past year.
• 66% were concerned about data poisoning and sensitive data leaking through training datasets.

👉 Read Novee's analysis of how AI red teaming is reshaping modern defense strategies

Context

AI red teaming is the practice of testing AI systems the way an attacker would, with the aim of finding failures in reasoning, prompts, retrieval layers, and agent behaviour before those failures are exploited in production. The primary security gap is that traditional testing is designed for code and infrastructure, while AI systems can be manipulated through meaning, context, and conversational control. That makes AI red teaming a governance issue as much as a technical one, especially where models touch identity, secrets, or delegated access.

The article’s core claim is that one-time testing is no longer adequate because models drift, data changes, and agent workflows expand after deployment. For IAM and NHI programmes, that intersects directly with who or what can query data, trigger tools, and inherit privileges from connected systems. The result is a control problem that spans model risk, access governance, and operational monitoring, not just security review at launch.

Key questions

Q: How should security teams use AI red teaming results in production governance?

A: Security teams should treat red team results as control evidence, not as a finished deliverable. Every finding should map to a runtime policy, a logging requirement, or an enforcement gap that can be tracked to closure. If a weakness cannot be tied to a production control, the organisation has only identified exposure, not reduced it.

Q: Why do AI systems create governance gaps that standard app security misses?

A: AI systems create governance gaps because their behavior depends on prompts, model state, external data, and connected tools, not just static code. A standard application review may miss information disclosure, prompt manipulation, or workflow abuse because the risk emerges from how the system is used at runtime.

Q: What breaks when AI agents are given access without identity governance?

A: What breaks is accountability. The organisation may see actions, logs, and alerts, but it cannot reliably tie them to a governed identity with clear scope and revocation. That creates uncontrolled blast radius, especially when agents can reach sensitive systems through shared tokens, delegated service accounts, or broad API access.

Q: How do security teams know if AI red teaming is working?

A: AI red teaming is working when testing finds real prompt injection paths, over-scoped integrations, and policy gaps before attackers do, and when fixes are re-tested successfully after model or workflow changes. The strongest signal is repeatable reduction in exposed authority, not a lower number of red-team findings on its own.

Technical breakdown

Semantic attacks and prompt injection

Semantic attacks target the language and context processing layer of an AI system rather than exploiting code in the usual sense. Prompt injection, roleplay attacks, and hidden instructions in retrieved content can cause the model to ignore system prompts, disclose sensitive information, or follow an attacker’s instructions. In RAG environments, the model may trust retrieved text too much, which turns external content into a control bypass. The technical risk is not only output manipulation but also instruction hierarchy confusion, where user-controlled text outranks policy controls.

Practical implication: treat prompts, retrieved documents, and tool outputs as untrusted inputs and test them under adversarial conditions.

Data poisoning, leakage, and model extraction

Data poisoning changes model behaviour by corrupting training or fine-tuning inputs, while leakage occurs when the model memorises and later reveals sensitive data such as credentials or PII. Model extraction is different again: an attacker repeatedly queries the model to infer its internal logic or replicate its behaviour. These are governance failures as much as technical ones because they show that data lineage, training controls, and post-deployment monitoring must extend across the full model lifecycle. Without that, the model becomes a system whose behaviour can be shaped by hostile inputs.

Practical implication: add controls around training data provenance, leakage testing, and abuse-rate monitoring across model versions.

Agentic misuse and tool delegation

AI agents expand the attack surface because they can decide when to act, which tools to use, and how to chain steps toward a goal. That makes them distinct from passive LLMs. Once an agent can query databases, operate file systems, or call internal APIs, a successful prompt manipulation can become privilege misuse or unauthorized action. The security issue is not merely output quality. It is delegated execution. In identity terms, the agent behaves like a non-human identity with access that may exceed the task it was given.

Practical implication: bind each agent to least-privilege tool access, scoped approvals, and monitored execution boundaries.

Threat narrative

Attacker objective: The attacker wants to make the AI system reveal information, ignore policy boundaries, or execute actions that advance data theft, fraud, or internal compromise.

1. Entry begins with adversarial prompts, hidden instructions, or poisoned retrieval content that reaches the model through normal user or document channels.
2. Escalation occurs when the model accepts attacker-controlled context over its intended safety or policy instructions, or when an agent uses delegated tools beyond the intended task.
3. Impact follows when the system exposes sensitive data, executes unauthorized actions, or amplifies misuse such as phishing, fraud, or internal access abuse.

NHI Mgmt Group analysis

AI red teaming is no longer a specialist test harness, it is a governance control. The article shows that AI systems fail in ways traditional security tooling is not built to observe, especially where meaning, context, and delegation determine outcomes. That shifts the control question from whether a model is accurate to whether it is governable under adversarial pressure. Practitioners should treat red teaming as part of model risk and identity governance, not a one-off validation exercise.

Semantic attack surface: the real risk is that normal-looking text can drive abnormal system behaviour. This is especially important in RAG and agentic environments, where retrieved content and tool calls can become control inputs. That means policy enforcement has to sit both before and after the model, with clear trust boundaries around prompts, documents, and actions. The practitioner conclusion is that input filtering alone is not enough.

Agentic AI behaves like a non-human identity with execution rights. Once an AI system can choose tools, sequence actions, and interact with internal resources, privilege design becomes an identity problem. That intersection matters to NHI, PAM, and IAM teams because the agent may inherit access without a lifecycle model that matches its behaviour. Organisations need task-scoped entitlements, session-level logging, and explicit offboarding for AI agents just as they would for service accounts.

Continuous testing reflects the pace of model change, not a testing preference. The article correctly points out that updates, fine-tuning, and new data exposures can invalidate earlier results. That makes the weakest assumption in many AI programmes the idea that a passing assessment creates lasting assurance. The field needs continuous adversarial validation, measured drift detection, and clear ownership for when a model moves from safe enough to re-test required.

Named concept, model drift exposure window: this is the period after deployment when updates, retrieval changes, or new data sources reopen risks that a previous red team exercise may have closed. The concept matters because AI systems can become unsafe without any code change visible to traditional controls. The practitioner takeaway is that red team evidence must expire alongside the model state it tested.

What this signals

AI red teaming should now be treated as an adjacent control to NHI governance, not a separate discipline. As agents begin to call tools and traverse internal data sources, the identity of the system becomes part of the attack surface. That means security teams need to align red team findings with access reviews, secrets handling, and privilege boundaries, especially where models touch high-value data or privileged workflows.

Model drift exposure window is the operational gap most teams underestimate. A passing assessment only applies to one model state, one data set, and one workflow configuration. The programme implication is clear: tie adversarial testing to deployment events, retraining, and retrieval changes, and use 52 NHI Breaches Analysis to pressure-test how quickly control failures turn into real incidents.

The control model is moving toward continuous validation across model, data, and identity layers. Teams that rely on annual review cycles will miss the pace at which AI behaviour changes, especially when agents gain new tools or data access. Practitioners should prepare for evidence-driven governance, where red team results, privilege scope, and monitoring signals all feed the same risk decision process.

For practitioners

• Implement continuous AI red teaming Run adversarial tests after every material model, prompt, retrieval, or agent change, then feed findings into the deployment gate rather than treating them as a periodic audit.
• Scope trust boundaries around prompts and retrieval Classify user input, retrieved documents, and tool outputs as untrusted until validated, especially in RAG pipelines where hidden instructions can override intended behaviour.
• Bind agents to task-scoped privileges Limit each AI agent to the minimum tool set, dataset, and API scope needed for the task, then log every delegated action for review and rollback.
• Add leakage and poisoning checks to the model lifecycle Test for memorised secrets, prompt injection, and poisoned training data before release and after retraining, with explicit owners for remediation and retesting.

Key takeaways

• AI red teaming addresses a control gap that traditional security testing cannot see, because attackers can manipulate meaning, context, and delegated execution rather than code alone.
• The strongest signal in the article is operational, not theoretical: model updates, fine-tuning, and new data exposures continuously reopen risks that one-time testing cannot cover.
• For practitioners, the right response is continuous adversarial testing tied to identity, data, and tool-access governance across the full AI lifecycle.

Key terms

• Semantic Attack: An attack that manipulates meaning, context, or instructions rather than exploiting software code directly. In AI systems, the adversary aims to change the model’s behaviour through language, retrieved content, or conversation flow so the output or action becomes unsafe or unauthorized.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Model Drift: Model drift is the gradual change in a model’s behaviour or performance after deployment. It happens when the operating environment, user patterns, or inputs no longer match the conditions used to validate the system. Drift matters because a model can appear functional while no longer meeting approved standards.
• Non-Human Identity (NHI): A digital identity assigned to a non-human entity such as a software application, service account, API key, bot, machine, or AI agent that enables it to authenticate and interact with systems without direct human involvement. NHIs now outnumber human identities in most enterprises by 25 to 50 times.

What's in the full article

Novee's full article covers the operational detail this post intentionally leaves for the source:

• The practical red teaming workflow for models, prompts, RAG pipelines, and autonomous agents.
• The named tools and testing patterns the vendor recommends for continuous validation.
• Implementation guidance for turning findings into release gates and remediation workflows.
• The vendor's examples of how purpose-trained offensive models are used in testing.

👉 The full Novee article covers the attack techniques, testing lifecycle, and operational examples in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security practitioners connect identity control to the systems and workflows that now shape AI risk.