AI red teaming must test the full system, not just the model

At a glance

What this is: Terra Security argues that AI red teaming should assess the full deployed system, not just the model, because real risk emerges in prompts, retrieval, tools, permissions, and workflow layers.

Why it matters: For IAM, NHI, and AI governance teams, this matters because AI access decisions now span identities, permissions, and downstream actions, not just model safety or application testing.

By the numbers:

• 70% of organisations grant AI systems more access than they would give a human employee performing the exact same job.
• Only 44% of organisations have implemented any policies to manage their AI agents, despite 92% agreeing that governing AI agents is critical to enterprise security.
• Organisations that describe themselves as confident in their AI deployment actually experience a 72% security incident rate, compared to 33% for those who remain cautious.
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.

👉 Read terra's analysis of AI red teaming across the full deployed system

Context

AI red teaming is the practice of testing whether a deployed AI system can be manipulated into doing something harmful in its real operating context. In the context of AI red teaming, the key governance problem is that model-centric checks often ignore the identity, permission, and integration layers that determine whether an unsafe interaction becomes a real incident.

Modern AI systems behave less like standalone software and more like interconnected decision chains. That makes the attack surface bigger than the model itself, and it puts identity and access control squarely into scope because tools, data sources, and action paths all depend on governed permissions.

This is where security teams often overestimate the value of benchmark testing. A system can look safe in a lab and still be exploitable once retrieval permissions, tool access, and user context are combined, which is now the typical enterprise condition rather than an edge case.

Key questions

Q: How should security teams run AI red teaming for GenAI systems?

A: Start with the system’s trust boundaries, then test prompts, retrieval sources, tool calls, and output handling together. Good AI red teaming does not stop at finding bad answers. It checks whether the system can be pushed into revealing data, bypassing policy, or taking unsafe actions through connected interfaces.

Q: Why do AI systems require different security testing than traditional software?

A: AI systems can fail through interaction, retrieval, and probabilistic behaviour rather than only through code defects. A model may respond differently to hidden prompts, external content, or changing context, which makes static scanning insufficient. Security teams need adversarial testing because risk can emerge after deployment, not only during development.

Q: What do organisations get wrong about model safety evaluations?

A: They often assume that a model that refuses risky prompts is therefore well tested. In practice, refusal can prevent researchers from generating the scenarios needed to find bias, manipulation, and unsafe completions. Good evaluation measures whether the system can be probed under realistic adversarial conditions, not whether it politely declines abuse.

Q: How should security teams govern AI agents that can access enterprise systems?

A: Security teams should govern AI agents as non-human identities with explicit ownership, scoped privileges, and continuous monitoring. The control set should include inventory, task-bound credentials, audit trails, and revocation paths. If an agent can call tools or touch production systems, it belongs in the same governance model as service accounts and other machine identities.

Technical breakdown

Why model safety testing misses system-level AI risk

Model safety testing asks whether the model produces unsafe or policy-violating outputs under controlled prompts. AI red teaming asks a different question: whether the deployed system can be made to behave unsafely when the model is connected to retrieval, tools, memory, and business workflows. The distinction matters because most enterprise harm comes from the system translation layer, not the text generation model alone. A model may refuse direct abuse yet still leak data through a retrieval path, trigger a tool call, or expose system instructions through indirect injection. That is why the environment, not just the model, must be tested.

Practical implication: Test the full AI stack, including prompts, retrieval, tools, and workflow logic, before treating a model as production-ready.

How retrieval, tools, and permissions expand the AI attack surface

Retrieval-augmented generation introduces a trust boundary between what the model can see and what it should see. If retrieval scopes are too broad, an attacker can use crafted content or access paths to surface confidential or cross-tenant data. Tool and function calls create a second boundary, because model output can become an external action such as querying a system, changing a record, or invoking an API. That is where identity governance becomes critical: the permissions granted to the AI system determine whether a prompt becomes a harmless answer or an unauthorized action. In practice, the attack surface is defined by access, not by the model alone.

Practical implication: Map every AI tool and retrieval path to explicit permissions and review whether the AI can act beyond the minimum required scope.

Why agent and workflow manipulation is a governance problem

Agentic AI changes the risk profile because the system can sequence actions over time, not just answer a prompt. When an AI agent can retrieve data, choose tools, and trigger workflow steps, security teams are no longer governing a conversational interface. They are governing an actor with delegated operational reach. That makes access review, ownership, logging, and escalation boundaries central controls. If those controls are weak, the organisation may not know whether an AI system accessed the right data for the right reason or completed a chain of actions that should have been blocked. This is a governance gap as much as a technical one.

Practical implication: Apply identity, logging, and approval controls to agent workflows so every action chain is attributable and bounded.

Threat narrative

Attacker objective: The attacker wants to turn a trusted AI workflow into a path for data exposure, unauthorized action, or broader system abuse.

1. Entry begins when an attacker or malicious content reaches the AI system through a user-facing interface, retrieved document, or connected integration.
2. Escalation occurs when prompt injection, retrieval abuse, or tool manipulation causes the system to reveal restricted data or invoke actions outside its intended scope.
3. Impact follows when the compromised AI workflow exfiltrates sensitive information, triggers unauthorized business actions, or creates downstream security harm.

NHI Mgmt Group analysis

AI red teaming is now an identity governance problem as much as a testing discipline. Once AI systems can access tools, data sources, and business workflows, the question is no longer only whether the model is safe. It is whether delegated access is scoped tightly enough to prevent a harmless prompt from becoming a harmful action. That intersection between AI behaviour and permission design is where IAM, PAM, and NHI governance now converge. Practitioners should treat AI red teaming findings as access-control evidence, not just model-quality feedback.

Context is the new control surface. The article is right to emphasise that the full deployment context determines exploitability. Model-only evaluations can miss the trust boundaries created by retrieval layers, system prompts, memory, and tool integrations, which means the real governance unit is the deployed system, not the model artifact. This broadens the work for security teams and pushes AI assurance into the same operational mindset used for NHI and privileged access governance. Practitioners should inventory context paths before they attempt to score risk.

Blast-radius control is the core design principle for AI systems. The most useful named concept here is blast-radius control, which means limiting what an AI system can access and do even when its behaviour is manipulated. That principle is familiar in zero trust and privileged access design, but AI systems make it harder because one interface can conceal many back-end permissions. The practical lesson is that safe behaviour in one test does not prove safe impact in production. Practitioners should measure the consequences of misuse, not just the frequency of refusals.

Continuous validation will matter more than one-time assessment. AI applications change too quickly for annual testing to be sufficient, especially when prompts, retrieval content, tools, and permissions evolve separately. This is why continuous red teaming is becoming part of the operating model rather than a pre-launch gate. For the wider field, that signals a shift from static assurance to ongoing control verification. Practitioners should build recurring testing into release, change, and access review cycles.

The governance gap is between what developers intend and what the system can actually do. Many AI programmes still assume that a model refusal or safe benchmark score indicates acceptable risk. That assumption breaks once business logic, tool access, and identity boundaries are introduced. The article shows that enterprise AI risk emerges from the gap between intended behaviour and operational permissions. Practitioners should align AI risk ownership across AppSec, IAM, and platform teams before that gap turns into an incident.

What this signals

Blast-radius control: AI programmes now need the same discipline applied to privileged access, with scope limits, monitoring, and rollback paths defined before deployment. If an AI system can reach tools or data it should not touch, the governance model is already failing, even if the model itself appears safe.

The strongest operational signal is no longer whether the model refuses bad prompts. It is whether the organisation can prove that retrieval paths, tool permissions, and delegated actions remain within approved boundaries as changes ship. That makes AI assurance a change-management problem as much as a testing problem, which aligns closely with the controls described in NIST SP 800-207 Zero Trust Architecture and NIST SP 800-53 Rev 5 Security and Privacy Controls.

As AI adoption expands, security teams should expect more findings that look like identity failures, not just model failures. The programme implication is clear: AI red teaming, NHI governance, and privilege review need to converge into one control plane for access, action, and accountability.

For practitioners

• Map AI access to every back-end permission Inventory each AI system's retrieval sources, tools, APIs, and workflow permissions, then compare them with the minimum access required for the business task. Pay special attention to delegated actions that can trigger writes, approvals, or external calls.
• Test the whole AI deployment, not just prompts Include system prompts, memory, retrieval pipelines, tool execution, and user-facing interfaces in every red team scope. A model refusal is not evidence that the surrounding system cannot be manipulated into harmful behaviour.
• Treat agent workflows as privileged pathways Assign ownership, logging, and escalation thresholds to AI agent actions the same way you would for privileged human workflows. If an agent can change records or invoke external systems, the action path needs traceability and boundary checks.
• Build continuous validation into change management Re-test AI systems whenever models, prompts, retrieval content, or permissions change. That includes regression tests for tool abuse, retrieval leakage, and indirect prompt injection, because those risks often appear after configuration drift rather than initial deployment.

Key takeaways

• AI red teaming is about whether a deployed system can be manipulated into real operational harm, not whether a model passes a safety benchmark.
• The evidence points to a widening gap between AI access and AI governance, especially where delegated tools and retrieval layers create hidden blast radius.
• Security teams should treat AI red teaming findings as access-control issues and build continuous validation into change management.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Retrieval-augmented Generation: Retrieval-augmented generation is a pattern where an AI model pulls external information before generating output. The security challenge is that access rules can weaken when data is chunked, embedded, cached, or reused, so source permissions may not automatically follow the content into the model's context.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Agent Workflow: An agent workflow is a machine-driven sequence where an AI system can call tools, access data, and continue execution with limited human involvement. These workflows need governance that sees API activity and tool use, not just browser sessions or user clicks.

What's in the full article

terra's full article covers the operational detail this post intentionally leaves for the source:

• Step-by-step scope guidance for testing prompts, retrieval layers, tool calls, and workflow logic together.
• Practical examples of how attackers abuse indirect prompt injection, memory poisoning, and function-call chains.
• A maturity model for embedding AI red teaming into continuous validation and remediation workflows.
• Implementation context for security, AppSec, and red team teams that need to operationalise the testing programme.

👉 terra's full post covers scope design, attack dimensions, and continuous validation practices

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It gives security practitioners a structured way to connect identity controls to broader operational risk.