TL;DR: Automated AI red teaming, with Repello AI reporting it, uses autonomous agents to map attack surfaces, plan multi-turn exploits, and validate findings faster than manual testing can sustain. Agentic systems then need humans to sort breadth from severity and turn findings into remediation.
Editorial analysis by NHI Mgmt Group, based on content published by Repello AI: “Automated AI Red Teaming: How Autonomous Agents Test AI Security at Scale”.
Key questions
Q: What breaks when AI red teaming is treated like traditional penetration testing?
A: Teams miss the AI-specific attack surface, especially prompt handling, output manipulation, retrieval poisoning, and tool misuse.
Q: How do security teams know if AI red teaming is working?
A: AI red teaming is working when testing finds real prompt injection paths, over-scoped integrations, and policy gaps before attackers do, and when fixes are re-tested successfully after model or workflow changes.
Q: Should organisations combine AI red teaming with traditional manual testing?
A: Yes. Combine both because they expose different failure modes. AI red teaming is useful for fast, large-scale adversarial probing of model behavior, prompt injection, jailbreaks, data leakage, and unsafe tool use. Traditional manual testing catches workflow gaps, privilege misuse, edge cases, and control failures across IAM, NHI, cloud, and application layers. Together they give broader coverage, better prioritisation, and a more realistic view of operational risk.
Practitioner guidance
- Map the AI attack surface before testing Inventory system prompts, tools, retrieval sources, input modes, and output paths before any automated engagement so the test scope matches the real architecture.
- Use scripted and agent-driven testing together Run deterministic payload libraries for regression and scheduled agent-driven runs for deeper chain discovery, especially after prompt, tool, or model changes.
- Set authorisation boundaries for tool testing Document which tools, APIs, and environments an automated tester may touch, and keep production access explicit rather than assumed.
Bottom line: Automated AI red teaming addresses the scale problem created by fast-changing prompts, tools, and retrieval sources in agentic systems.
Glossary terms: Automated AI Red Teaming, Agent-Driven Testing, Attack Surface Recon, Business Logic Exploitation
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
Automated AI red teaming is becoming the practical control layer for agentic systems. Manual assessments still matter, but they cannot keep pace with weekly prompt changes, new tools, and expanding retrieval corpora. The governance problem is not whether a team can test an AI system, but whether it can test it repeatedly enough to stay relevant. Practitioners should treat automated red teaming as part of continuous validation, not as a one-off specialist exercise.
A few things that frame the scale:
- Only 13% of organisations feel extremely prepared for the reality of agentic AI despite the majority racing toward autonomous adoption, according to the 2026 Infrastructure Identity Survey.
- Claude Code-assisted commits leaked secrets at a rate of 3.2%, more than double the human-only baseline of 1.5%, with peaks reaching 31 secrets per 1,000 commits in August 2025, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: When should organisations prioritise agent-driven testing over scripted checks?
A: Prioritise agent-driven testing when the AI system uses tools, retrieval, or multi-step workflows that can be abused through chained interactions. Scripted checks remain useful for fast regression and known patterns, but agent-driven testing becomes the better investment when the target's architecture is changing, complex, or likely to hide novel failure modes.
👉 Read our full editorial: Automated AI red teaming closes the coverage gap in agentic systems