AI red teaming is exposing the limits of traditional security

At a glance

What this is: This article argues that AI systems create attack paths that traditional security testing misses, especially through semantic manipulation, poisoned data, and drifting model behaviour.

Why it matters: It matters because IAM, PAM, and broader security teams must govern not only access to AI systems but also the way models, prompts, and agent workflows can be abused after deployment.

By the numbers:

• 97% of organisations surveyed had experienced security issues tied to Gen AI in the past year, according to Capgemini research cited by Novee.
• 66% were concerned about data poisoning and sensitive data leaking through training datasets, according to Capgemini research cited by Novee.

Context

AI red teaming is a structured adversarial testing method for models, prompts, retrieval systems, and agent workflows. Traditional security testing is strong at finding code flaws, but it does not reliably expose behavioural failures where a system is manipulated through language, context, or poisoned inputs.

The governance gap is that AI systems are not static assets. They change as models are updated, fine-tuned, retrained, and exposed to new data, which means a control that passed last month can fail today. For IAM and NHI programmes, the intersection is real wherever AI systems have tool access, retrieval access, or delegated permissions.

Key questions

Q: How should security teams use AI red teaming results in production governance?

A: Security teams should treat red team results as control evidence, not as a finished deliverable. Every finding should map to a runtime policy, a logging requirement, or an enforcement gap that can be tracked to closure. If a weakness cannot be tied to a production control, the organisation has only identified exposure, not reduced it.

Q: Why do AI systems need continuous security testing after launch?

A: AI systems change over time as models, prompts, integrations, and access paths evolve. That creates new attack paths even when the original release was vetted. Continuous testing helps teams find drift, newly exposed flaws, and abuse opportunities before attackers do. For regulated or high-impact systems, this is part of maintaining secure by design rather than a one-time control.

Q: What are the signs that an AI model is too risky to deploy?

A: Warning signs include a high jailbreak failure rate, weak prompt injection resistance, frequent hallucinations, unsafe content generation, and unstable responses when prompts are manipulated. If a model can be pushed into revealing sensitive data, producing harmful outputs, or behaving inconsistently across tests, it should be treated as unsuitable for enterprise use until controls and retesting show measurable improvement.

Q: How should organisations govern external tools used by AI agents?

A: Organisations should review external tools as security inputs, not convenience features. Each tool needs ownership, approval, metadata inspection, and ongoing monitoring for hidden instructions or unexpected behaviour. If an AI agent can act on a tool, then the tool’s provenance and control status should be governed like any other sensitive integration.

Technical breakdown

Why semantic attacks bypass conventional security controls

Semantic attacks target meaning, not syntax. A prompt can look harmless to a scanner while still instructing a model to reveal hidden data, ignore policies, or produce unsafe output. This is different from malware or SQL injection, where the payload has a recognizable technical signature. In AI systems, the failure often sits in the instruction hierarchy, retrieval context, or safety alignment layer. That is why model evaluation and standard application security checks are necessary but insufficient on their own.

Practical implication: test the model’s decision boundary, not just its surrounding infrastructure.

How model drift turns one-time validation into a weak control

Model drift is the way AI behaviour changes after deployment because of new weights, updated prompts, fresh retrieval data, fine-tuning, or altered tool access. The important point is that risk is dynamic, not point-in-time. A model that resisted a jailbreak during pre-launch testing may later respond differently after a retraining cycle or a data source change. Continuous red teaming exists because the attack surface is moving, and security evidence ages quickly in AI environments.

Practical implication: re-test after every material model, data, or workflow change.

Why agent workflows create identity and privilege risk

Once an AI system can call tools, query APIs, or act on behalf of a user, it becomes a delegated actor with real permissions. That changes the threat from output manipulation to action abuse. Goal hijacking, permission escalation, and misuse of connected resources become possible when the agent can chain decisions across systems. In identity terms, the question is no longer only whether the model is safe, but whether its issued permissions are scoped, observable, and revocable at the right point in the workflow.

Practical implication: govern agent permissions as you would any other high-risk non-human identity.

NHI Mgmt Group analysis

AI red teaming has become a control validation discipline, not a specialist exercise. The article shows that AI risk is no longer confined to research labs or model teams, because attackers can work through prompts, retrieved content, and agent workflows. That makes continuous adversarial testing a governance requirement for organisations deploying models into production. The practitioner conclusion is straightforward: if the AI can reason or act, it must also be tested under adversarial conditions.

Model drift is the governance failure mode most organisations underestimate. AI systems do not stay in the state they were in at launch, and that breaks the assumption behind one-time approval gates. A model can move from safe to unsafe because of new training data, updated prompts, or altered tool access. The practitioner conclusion is that evidence of safety must be time-bound and revalidated after every material change.

Agentic AI identity is now part of the security problem. Once a model can query APIs, retrieve documents, or perform operations on a user’s behalf, it behaves like a non-human identity with delegated authority. That means access scope, revocation, and observability matter as much as prompt safety. The practitioner conclusion is to treat AI agents as governed identities, not just as software features.

Purpose-trained offensive models will reshape how red teams work. The article’s comparison between general-purpose LLMs and adversarially trained models points to a broader market shift: security teams need tools designed for hostile reasoning, not fluent assistance. This does not replace human judgment, but it does change how testing scales and where evidence is generated. The practitioner conclusion is to align offensive testing capability with the attack surface, not the convenience of the model.

Semantic attack resilience is the right named concept for this problem. The article repeatedly shows that the real break point is not code execution but meaning manipulation. That includes jailbreaks, hidden instructions, retrieval poisoning, and social engineering through language. The practitioner conclusion is to measure whether a model can resist adversarial meaning, not only whether the surrounding system resists technical exploits.

What this signals

Semantic attack resilience: organisations now need a control objective that measures whether a model can withstand meaning manipulation, not just conventional exploitation. That shifts AI security from static assessment to continuous adversarial assurance, which is closer to how identity and access governance already treats privileged change.

The operational implication for practitioners is that AI releases should inherit the same discipline as privileged workflows. If the model, its retrieval layer, or its agent permissions change, the security evidence is stale and the test cycle has to restart.

For practitioners

• Define adversarial test scope for AI systems Specify whether the red team is testing the model, prompt layer, retrieval system, agent workflow, or the full stack. Make success criteria measurable, such as resistance to PII disclosure or unauthorised tool use across repeated trials.
• Retest after every material model change Treat model updates, fine-tuning, new retrieval sources, and tool permission changes as security events. Re-run adversarial tests after each change rather than relying on launch-time approval.
• Build runtime monitoring for behavioural anomalies Look for unexpected policy overrides, abnormal retrieval patterns, repeated refusal bypass attempts, and agent actions that drift from intended objectives. Tie alerts to containment steps before the model can complete the task.
• Constrain agent permissions to task-scoped access Limit APIs, file system operations, and database queries to the minimum needed for the workflow. Review whether the agent’s permissions can be revoked cleanly when the task ends or the context changes.
• Feed findings back into alignment and guardrails Convert every confirmed weakness into a reproducible test case, then update policy layers, retrieval filtering, and output handling rules so the same behaviour fails consistently in future runs.

Key takeaways

• AI systems fail in ways that conventional security controls do not reliably detect, especially when attackers manipulate meaning, retrieval context, or delegated actions.
• Model behaviour changes over time, so launch-time validation is not enough to establish ongoing trust in production AI systems.
• Red teaming has to cover prompts, retrieval, and agent permissions together if organisations want to reduce misuse, leakage, and unsafe actions.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Semantic Attack: An attack that manipulates meaning, context, or instructions rather than exploiting software code directly. In AI systems, the adversary aims to change the model’s behaviour through language, retrieved content, or conversation flow so the output or action becomes unsafe or unauthorized.
• Model Drift: Model drift is the gradual change in a model’s behaviour or performance after deployment. It happens when the operating environment, user patterns, or inputs no longer match the conditions used to validate the system. Drift matters because a model can appear functional while no longer meeting approved standards.
• Agentic workflow: An agentic workflow is a sequence of tasks executed by an AI agent with some level of tool access and decision authority. In security terms, the workflow matters because it can span multiple systems, identities, and permissions, which makes attribution and revocation harder than with ordinary automation.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is designed for practitioners who need to connect delegated access, lifecycle control, and governance across human and machine identities.