Notifications
Clear all
Topic starter
02/08/2026 11:42 am
TL;DR: AI red teaming evaluates the full deployed AI system, including prompts, retrieval layers, tools, agents, permissions, and workflows, because model safety checks alone miss exploitable paths that can lead to data exposure or unauthorized actions, according to Terra Security. The real governance shift is from testing outputs in isolation to validating whether attackers can turn AI behaviour into operational impact.
NHIMG editorial — based on content published by terra: AI Red Teaming: What Security Teams Need to Know
By the numbers:
- 70% of organisations grant AI systems more access than they would give a human employee performing the exact same job.
- Only 44% of organisations have implemented any policies to manage their AI agents, despite 92% agreeing that governing AI agents is critical to enterprise security.
- Organisations that describe themselves as confident in their AI deployment actually experience a 72% security incident rate, compared to 33% for those who remain cautious.
Questions worth separating out
Q: How should security teams run AI red teaming for GenAI systems?
A: Start with the system’s trust boundaries, then test prompts, retrieval sources, tool calls, and output handling together.
Q: Why do AI systems require different security testing than traditional software?
A: AI systems can fail through interaction, retrieval, and probabilistic behaviour rather than only through code defects.
Q: What do organisations get wrong about model safety evaluations?
A: They often assume that a model that refuses risky prompts is therefore well tested.
Practitioner guidance
- Map AI access to every back-end permission Inventory each AI system's retrieval sources, tools, APIs, and workflow permissions, then compare them with the minimum access required for the business task.
- Test the whole AI deployment, not just prompts Include system prompts, memory, retrieval pipelines, tool execution, and user-facing interfaces in every red team scope.
- Treat agent workflows as privileged pathways Assign ownership, logging, and escalation thresholds to AI agent actions the same way you would for privileged human workflows.
What's in the full article
terra's full article covers the operational detail this post intentionally leaves for the source:
- Step-by-step scope guidance for testing prompts, retrieval layers, tool calls, and workflow logic together.
- Practical examples of how attackers abuse indirect prompt injection, memory poisoning, and function-call chains.
- A maturity model for embedding AI red teaming into continuous validation and remediation workflows.
- Implementation context for security, AppSec, and red team teams that need to operationalise the testing programme.
👉 Read terra's analysis of AI red teaming across the full deployed system →
AI red teaming and the governance gap teams are missing?
Explore further
02/08/2026 12:54 pm
AI red teaming is now an identity governance problem as much as a testing discipline. Once AI systems can access tools, data sources, and business workflows, the question is no longer only whether the model is safe. It is whether delegated access is scoped tightly enough to prevent a harmless prompt from becoming a harmful action. That intersection between AI behaviour and permission design is where IAM, PAM, and NHI governance now converge. Practitioners should treat AI red teaming findings as access-control evidence, not just model-quality feedback.
A question worth separating out:
Q: How should security teams govern AI agents that can access enterprise systems?
A: Security teams should govern AI agents as non-human identities with explicit ownership, scoped privileges, and continuous monitoring. The control set should include inventory, task-bound credentials, audit trails, and revocation paths. If an agent can call tools or touch production systems, it belongs in the same governance model as service accounts and other machine identities.
👉 Read our full editorial: AI red teaming must test the full system, not just the model
