AI red teaming must test the full system, not just the model

At a glance

What this is: AI red teaming is presented as a full-system adversarial test for deployed AI, with the key finding that model-only checks miss exploitable paths in retrieval, tools, permissions, and workflows.

Why it matters: IAM, AppSec, and AI governance teams need to treat AI systems as operational attack surfaces, because the real risk sits in how models interact with data, tools, and authorization boundaries.

Context

AI red teaming is the practice of attacking a live AI system the way an adversary would, rather than scoring a model against static benchmarks. The article argues that model behaviour alone does not reveal how prompts, retrieval layers, tools, and workflow logic combine into a real attack surface, which is why a gateway chat test is not enough.

That matters for identity and governance because AI systems often inherit access, permissions, and decision authority from the surrounding application. Once an AI can retrieve data, call tools, or trigger business actions, the question is no longer whether the model is safe in isolation, but whether the deployed system can be pushed beyond its intended privilege and operational scope.

Key questions

Q: How should security teams test AI agents that can call tools and APIs?

A: They should test the agent's actual execution paths, not just its text responses. That means covering tool calls, protocol handling, approvals, and state transitions with adversarial scenarios that mirror real misuse. If the test cannot reach the access point where the agent acts, it does not meaningfully assess the security risk.

Q: Why is model safety not enough to secure enterprise AI systems?

A: Model safety constrains behaviour, but it does not govern access. A safe-sounding model can still read data, call APIs, or change state if the surrounding system grants those permissions. Security teams need runtime authorisation, continuous identity verification, and audit trails that sit outside the model layer.

Q: What are the signs that an agentic AI red team is too narrow?

A: A red team is too narrow when it focuses mainly on prompt injection scores, jailbreaks, or single-turn harmful output tests. That approach misses persistence, tool selection errors, inter-agent trust abuse, and delayed attacks that unfold over time. Another warning sign is when the team cannot explain how it would test email, memory stores, APIs, or business logic failures.

Q: How should organizations approach the governance of AI agents?

A: Organizations should adopt a governance framework that incorporates continuous visibility, adaptive IAM practices, and stringent policy-based controls. This ensures that all agent actions are tracked, authorized appropriately, and assessed for compliance.

Technical breakdown

Why model safety testing misses the deployed AI attack surface

Model safety evaluations ask whether a model produces unsafe or policy-violating outputs under controlled prompts. That is useful, but it does not test the system that actually runs in production. Deployed AI applications add system prompts, retrieval pipelines, tool connectors, session memory, output handlers, and business logic, each of which can change the security outcome. A model can refuse obvious abuse and still be manipulated through indirect instructions, trusted content, or chained interactions that never appear in a benchmark. The security problem is therefore architectural, not just behavioural.

Practical implication: test the production AI stack as a whole, not only the model response layer.

How retrieval, tools, and permissions create AI system risk

RAG and tool-enabled systems create a translation layer between language output and real-world action. Retrieved content can introduce malicious instructions or unauthorized data, while tool calls can convert a persuasive response into a side effect in another system. If the permission model is too broad, the AI inherits more reach than the user or workflow intended. That is why AI red teaming has to probe access boundaries, tenant isolation, and whether the model can be induced to request or trigger actions it should never have had. In practice, exploitability emerges when language understanding meets over-permissioned execution.

Practical implication: validate retrieval scoping, tool authorization, and tenant boundaries before an AI workflow goes live.

Why agent and workflow manipulation changes the control model

AI agents are not just chat interfaces with better prompts. When a system can plan, call tools, or continue across multiple steps, it becomes possible to steer it into task completion that was never explicitly approved. Memory and context poisoning extend that risk by letting prior interactions influence later decisions. In governance terms, this means security cannot stop at content moderation or one-time prompt testing. The control point moves to the full decision chain, including what the system is allowed to remember, what it may do autonomously, and how downstream actions are authorized and audited.

Practical implication: govern autonomous actions, session memory, and auditability as first-class controls in AI deployments.

NHI Mgmt Group analysis

AI red teaming is now a system governance problem, not a model scoring exercise. The article correctly shifts attention away from isolated output checks and toward the deployed stack that includes retrieval, tools, permissions, and workflow logic. That is the right frame because operational harm emerges when model behaviour crosses into action, not when it merely produces text. Practitioners should treat the full system boundary as the security unit of analysis.

AI systems create an identity and privilege problem as soon as they can act on behalf of users or workflows. The identity bridge is not optional here: if an AI can access data or trigger tools, then its access path must be governed like any other high-risk non-human actor. That makes authorization, tenant separation, and audit trails central to AI security, not adjacent concerns. Practitioners should align AI controls with identity governance from the start.

Full-system red teaming exposes the blast radius that model benchmarks cannot see. A model may pass safety tests and still leak system prompts, retrieve unauthorized content, or chain tool calls into harmful outcomes. Blast-radius validation: that is the specific concept this article sharpens, and it should become a standard expectation for AI assurance. Practitioners should validate what the system can cause, not just what the model can say.

Continuous testing is the only credible operating model for changing AI environments. The article is right that one-off assessments age quickly when models, knowledge bases, workflows, and permissions keep changing. The governance implication is that AI red teaming belongs inside the change lifecycle, with findings routed to owners and reassessed as the system evolves. Practitioners should replace periodic checkbox reviews with continuous validation.

Model-only trust assumptions will fail wherever AI is allowed to connect to real business systems. The article shows that the dangerous part of AI is not the generation step but the connection between generated intent and executed action. That makes the surrounding controls, especially authorization and workflow guardrails, the decisive layer. Practitioners should govern AI as an execution environment, not a content engine.

What this signals

Blast-radius validation: this is the control idea practitioners should carry forward from the article. If the AI stack can retrieve, call tools, or trigger business workflows, then assurance has to measure the consequences of abuse, not just the likelihood of a bad prompt. That changes how security teams define pass or fail for AI review.

AI red teaming also has an identity consequence: once a system can act through tools and data sources, its permissions become part of the threat surface. Teams should align AI access with least privilege, tight scoping, and auditable delegation, or they will inherit the same governance failures that have long plagued over-permissioned service accounts.

For practitioners

• Scope red team tests to the full AI deployment Include the model, system prompt, retrieval pipeline, tools, APIs, memory, output handling, and workflow logic in every assessment so hidden attack paths are visible.
• Test authorization boundaries for AI access and action Verify that the AI cannot cross user-level, role-level, or tenant-level permissions when retrieving data or triggering downstream actions.
• Probe indirect prompt injection and retrieval abuse Use attacker-controlled content, retrieved documents, and third-party inputs to see whether they can override instructions or surface unauthorized data.
• Review tool-call governance before production Constrain which tools the AI may invoke, define approval boundaries for high-impact actions, and log every triggered call for investigation.
• Make red teaming continuous across change events Re-test whenever models, knowledge bases, workflows, or permissions change so new exploit paths are caught before they reach users.

Key takeaways

• AI red teaming has to cover the deployed system, because model-only testing leaves retrieval, tool, and workflow abuse untouched.
• The real failure mode is not unsafe text generation by itself, but the conversion of model behaviour into unauthorized data access or action.
• Security teams should govern AI permissions and red team continuously, because changing models, data, and workflows keep reopening the attack surface.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Retrieval-augmented Generation: Retrieval-augmented generation is a pattern where an AI model pulls external information before generating output. The security challenge is that access rules can weaken when data is chunked, embedded, cached, or reused, so source permissions may not automatically follow the content into the model's context.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Tool-call abuse: Tool-call abuse occurs when an agent or related workflow manipulates a tool invocation so the connector performs an unsafe action. The risk is technical and governance-related at once, because the call may look legitimate while still producing unauthorised access, data exposure, or workflow execution.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and agentic AI identity. It helps security teams connect access control, delegation, and lifecycle governance to the systems they are now being asked to protect.