Notifications
Clear all
Topic starter
02/08/2026 12:09 pm
TL;DR: AI red teaming must move from one-off validation to continuous testing because real-world AI flaws are increasingly triggered by prompt manipulation, data poisoning, model extraction, and tool abuse, according to Akto’s guide and cited research. Probabilistic behaviour changes the attack surface, so resilience depends on repeatable adversarial testing, not standard penetration testing alone.
NHIMG editorial — based on content published by Akto: AI Red Teaming for Agentic AI Security
By the numbers:
- 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).
- 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate.
Questions worth separating out
Q: How should security teams run AI red teaming for GenAI systems?
A: Start with the system’s trust boundaries, then test prompts, retrieval sources, tool calls, and output handling together.
Q: Why do agentic AI systems require continuous red teaming?
A: Agentic systems change when models are retrained, prompts are updated, retrieval sources shift, or new integrations are added.
Q: What breaks when AI red teaming is treated like traditional penetration testing?
A: Teams miss the AI-specific attack surface, especially prompt handling, output manipulation, retrieval poisoning, and tool misuse.
Practitioner guidance
- Scope the AI system as a governed identity surface Document every model, connector, retrieval source, API token, and workflow the agent can reach.
- Run manual red teaming before automating regression tests Use human testers to discover novel prompt injection, jailbreak, and tool-abuse patterns first, then convert the highest-value cases into repeatable test suites for release gates and production monitoring.
- Treat agent permissions like privileged access Apply least privilege to tool calling, data retrieval, and delegated actions.
What's in the full article
Akto's full blog covers the operational detail this post intentionally leaves for the source:
- Step-by-step scoping of AI red team engagements across prompts, integrations, data sources, and lifecycle stages.
- Examples of manual versus automated attack patterns, including discovery, escalation, persistence, and validation workflows.
- Guidance on tracking prompts, system state, and remediation evidence across iterative model changes.
- Practical discussion of governance, legal, and compliance coordination for AI red team findings.
👉 Read Akto's guide to continuous AI red teaming for agentic systems →
AI red teaming for agentic systems: are your controls keeping up?
Explore further
02/08/2026 1:13 pm
Continuous AI red teaming is now an identity governance control, not only a model safety exercise. Once an AI system can call tools or touch internal data, it behaves like a governed non-human identity with delegated access. That shifts the question from whether the model is safe in the abstract to whether its action boundary is enforceable, reviewable, and revocable. Practitioners should treat red teaming as part of access governance for agentic systems.
A question worth separating out:
Q: Who is accountable when an AI agent accesses sensitive data it was not meant to use?
A: Accountability sits with the team that approved the agent, its connectors, and its policy boundaries, not with the runtime behaviour alone. Organisations need ownership for intent, permissions, monitoring, and validation so they can prove whether the agent stayed inside its approved purpose. Without that, audit and regulatory response become retrospective guesswork.
👉 Read our full editorial: AI red teaming for agentic systems needs continuous testing
