AI red teaming shows why deterministic security no longer fits

At a glance

What this is: Lakera argues that AI red teaming must move from one-off testing to continuous evaluation because non-deterministic systems change behaviour with prompts, context, tools, and model updates.

Why it matters: IAM, PAM, and NHI teams need to treat AI application behaviour as a governed security surface, because access controls on their own do not stop prompt-driven misuse once the system can act.

Context

AI red teaming is the practice of stress-testing a system to see how it behaves under adversarial pressure. In this article, the core problem is not model quality alone but the fact that AI behaviour is non-deterministic, so the same application can respond differently as prompts, context, tool permissions, and model updates change.

That matters for identity governance because AI applications increasingly sit behind credentials, tool permissions, and action rights. A security programme that assumes stable, repeatable behaviour will miss failures that only appear when the system is asked to decide, delegate, or execute in a live workflow.

Key questions

Q: What breaks when AI red teaming is treated like traditional penetration testing?

A: Teams miss the AI-specific attack surface, especially prompt handling, output manipulation, retrieval poisoning, and tool misuse. A standard pen test can prove infrastructure exposure, but it cannot reliably measure whether the model will follow malicious instructions or misuse delegated access. That leaves a major assurance gap for agentic systems.

Q: Why do AI prompts create identity and data-security risk?

A: AI prompts create risk because they can carry sensitive content outside the original system’s protection boundary. Copy-paste and file uploads can move secrets, regulated data, or operational context into tools that may log, reuse, or connect that information elsewhere. The issue is not the prompt itself, but the identity and data path it opens.

Q: How do security teams know if AI red teaming is working?

A: AI red teaming is working when testing finds real prompt injection paths, over-scoped integrations, and policy gaps before attackers do, and when fixes are re-tested successfully after model or workflow changes. The strongest signal is repeatable reduction in exposed authority, not a lower number of red-team findings on its own.

Q: What should organisations do when AI systems can send emails or write to databases?

A: Treat those capabilities as delegated actions with explicit approval boundaries, logging, and review. Once an AI system can act in the environment, red teaming must test how it can be steered into using those permissions in unsafe ways. The question is not just what it generates, but what it can cause to happen.

Technical breakdown

Why prompts behave like code in AI systems

In non-deterministic AI systems, the prompt is not just input text. It becomes a control surface that can alter meaning, trigger tool use, and change downstream actions without changing the underlying infrastructure. That is why traditional perimeter thinking fails: the attacker is no longer only attacking the code path, but the semantic path. Static filters also struggle because malicious intent can be expressed in valid natural language. For security teams, the key technical shift is that language itself becomes part of the execution model, so testing must examine how prompts interact with tools, data, and permissions in combination.

Practical implication: test the full prompt-to-action path, not just the model response.

Why static benchmarks miss emergent AI failures

Traditional assurance assumes that a fixed test suite can represent the system. In AI applications, that assumption breaks because behaviour emerges through interaction, not just predefined inputs. A model can appear safe in one evaluation and fail later after a silent model update, a new tool integration, or a subtle change in context. This is not a quality defect in the conventional sense. It is an exposure to behavioural drift. Red teaming therefore has to become continuous, application-specific, and lifecycle-aware so that new failure modes are discovered after design, before release, and during monitoring.

Practical implication: run recurring evaluations whenever prompts, models, tools, or permissions change.

What changes when AI systems can take actions

Agentic AI changes the impact model because the system is no longer limited to generating text. Once it can write to databases, send emails, or execute code, prompt injection becomes an action-enabling threat rather than an output problem. That means the security question shifts from whether the model can be tricked to whether the application can be steered into making an unsafe decision. For governance, this creates a stronger need to inspect the tool-calling layer, the system prompt, and the final action path together, because the real risk is controlled execution under misleading instructions.

Practical implication: review tool permissions and execution paths as part of AI red teaming.

Breaches seen in the wild

• iOS apps leaking hard-coded secrets: Cybernews found 71% of 156,080 iOS apps leak hard-coded secrets, with open cloud storage and Firebase databases exposing user data.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Deterministic security assumptions collapse when the system can change behaviour at runtime. Traditional controls assume that the same input produces the same outcome, which is a poor fit for AI applications whose responses vary with phrasing, context, and model updates. That assumption fails because the application is no longer stable enough for point-in-time assurance to be meaningful. The implication is that identity and access programmes must stop treating AI behaviour as predictable by design and start treating it as a moving control surface.

Prompting is now an execution pathway, not a user interface concern. When the prompt influences tool selection, data access, or action timing, it becomes part of the security boundary. That creates a governance gap across human, NHI, and agentic workflows because the same application may be controlled by a person, backed by service credentials, and then executed by an autonomous component. Practitioners need a model for where language ends and authorization begins.

Continuous red teaming is the only realistic assurance model for non-deterministic systems. One-off testing produces a false sense of closure because AI systems drift after model updates, tool additions, and context changes. The named concept here is the runtime assurance gap: the distance between a passing evaluation and the next behavioural shift. Security teams should expect that gap to widen as AI applications become more operationally embedded.

Agentic AI turns misuse into delegated action, which raises the governance bar beyond model safety. Once a system can perform tasks, the relevant question is no longer only what it says but what it can be induced to do. That changes how teams think about access rights, monitoring, and accountability across the AI application lifecycle. Practitioners should reframe red teaming as control of behaviour under delegation, not just content moderation.

AI governance has to connect model assurance to identity and permission governance. The article’s central message is that tool access, data exposure, and execution rights are now inseparable from AI security. That makes AI red teaming relevant to IAM, PAM, and NHI governance because the security outcome depends on how authority is granted, bounded, and monitored. The practical conclusion is that AI assurance cannot live in isolation from identity controls.

What this signals

Runtime assurance gap: AI security programmes need a control model that assumes behavioural drift, because model updates and tool changes can invalidate a passing test without warning. That changes the operating rhythm from annual review to continuous verification.

As AI applications gain permission to act, the governance conversation has to join model safety, authorization scope, and monitoring into one programme. Otherwise, teams will keep evaluating text generation while the real risk sits in delegated execution.

For practitioners

• Map prompt-to-action pathways Identify where prompts influence tool selection, data access, or execution so red teaming can test the full decision chain, not only model output quality.
• Re-test after every behavioural change Schedule regression testing whenever the foundation model, system prompt, tools, or permissions change, because each change can alter the system’s security boundary.
• Scope red teaming to application context Test the specific AI workflow, business data, and allowed actions that matter in production instead of relying on generic jailbreak exercises.
• Treat autonomous actions as governance events Review database writes, emails, and code execution as controlled security actions that need explicit permission boundaries and monitoring.

Key takeaways

• AI red teaming is moving from point-in-time evaluation to continuous assurance because non-deterministic systems change with context, tools, and updates.
• The real security problem is no longer only model output quality. It is whether prompts can steer an application into unsafe tool use or delegated action.
• Identity and access teams need to govern AI behaviour as an execution surface, with red teaming, permission boundaries, and behavioural monitoring working together.

Key terms

• Non-deterministic system: A non-deterministic system does not produce reliably identical outcomes from identical inputs. In AI security, that means prompts, context, model updates, and tool connections can change the result, so assurance has to measure behaviour over time rather than assume fixed output patterns.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Behavioural Drift: Behavioural drift is the gradual change in what an identity does compared with what it was originally approved to do. For AI agents, drift can come from prompt changes, model updates, expanded integrations, or altered workflows, which makes access review alone an incomplete control.
• Continuous Red Teaming: Continuous Red Teaming is an ongoing practice of testing defenses by simulating realistic attacker behavior throughout the security lifecycle. It combines repeated adversarial exercises, monitoring, and feedback loops to expose weaknesses in people, processes, technology, and identity controls. The goal is to find exploitable gaps before real attackers do.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.