AI red teaming is shifting to persistent testing across agentic systems

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Mindgard's blog post: AI red teaming is moving from one-off model testing to continuous testing across agentic workflows, with the biggest risks now concentrated in tool use, memory, multimodal inputs, and supply chain exposure.

Why it matters: That matters because security teams now have to govern AI behaviour as an operational attack surface, not just a model output problem, and that intersects directly with identity, privilege, and delegated access controls.

By the numbers:

• fully automated jailbreak agents have achieved success rates as high as 97.14% in one study.
• more than 73% of tested LLM applications are vulnerable to prompt injection.
• indirect prompt injection attacks succeed only 0.5%-8.5% of the time.
• attack success rate jumps from 13% for single-turn attacks to 92% for multi-turn conversations.

Context

AI red teaming now addresses how models behave inside workflows, not just whether a single prompt can break a guardrail. The shift matters because systems that call tools, retain memory, and consume external content create attack paths that static testing misses, especially when the model is used in business processes rather than isolated demos.

The article’s core point is that the threat surface is expanding from prompt handling into orchestration, delegation, and runtime decision-making. That creates a direct governance bridge to agentic AI security because the control problem is no longer only model safety, but how inputs, permissions, memory, and tools are governed across the lifecycle.

Key questions

Q: How should security teams test generative AI systems for real-world abuse?

A: They should test the full execution path, not just the model prompt. That means probing retrieval, tool use, output handling, and any delegated actions the system can perform. The goal is to reproduce how an attacker would move from manipulation to impact, then capture whether the system leaks data, ignores guardrails, or performs unauthorised actions.

Q: Why do agentic AI systems create more security risk than standard chatbots?

A: Agentic systems can turn model output into action, which means a bad instruction can affect code flow, tool use, and downstream state. Standard chatbots usually stop at text generation. Agentic AI adds execution authority, so compromise can lead to disclosure, unauthorized changes, or misuse of credentials.

Q: What are the signs that voice AI red teaming is failing to keep pace?

A: Common warning signs are repeated unsafe completions, different behaviour after retraining, inconsistent handling of similar prompts, and security findings that reappear when conversation patterns change. If the same control only works in one model version or one prompt set, the testing programme is already behind the system’s actual behaviour.

Q: Should organisations prioritise prompt injection defence or tool permission control first?

A: Both matter, but tool permission control usually gives the faster risk reduction when agents can already call external systems. Prompt injection is the trigger, but excessive authority is what turns a successful injection into a harmful action. Reduce the permissions first, then harden the ingestion and retrieval paths that feed the agent.

Technical breakdown

Why agentic workflows create a larger attack surface

Agentic systems do more than generate text. They interpret inputs, choose tools, call APIs, retain context, and sometimes act across sessions. That means the security boundary is no longer the prompt alone but the chain of decisions and side effects that follow it. A successful injection can alter tool selection, workflow routing, memory state, or downstream actions, which turns a small input flaw into a broader runtime compromise. The core technical issue is that AI behaviour is conditional and stateful, so attack success can depend on session context, delegated permissions, and what the system is allowed to do once influenced.

Practical implication: Treat tool use, memory, and delegation as part of the attack surface, not just the model output.

Why multimodal and indirect prompt injection are hard to contain

Indirect prompt injection works by hiding instructions inside content the AI consumes, such as web pages, documents, images, or video frames. Multimodal attacks extend that pattern across text, vision, and structured documents, so the malicious instruction can survive outside normal prompt filters. The model may not distinguish trusted operator intent from adversarial content once the content enters the retrieval or reasoning chain. This is why content provenance, retrieval boundaries, and input sanitisation matter: the attack often succeeds before the model even generates an answer, because the poisoned material is already inside the decision path. See the Prompt Injection Guide.

Practical implication: Apply provenance checks and content filtering to every retrieval and ingestion path the model can read.

Why supply chain and permission scope now matter to AI security

AI supply chain risk now includes third-party models, integrations, automation pipelines, and the credentials those components inherit. If an agent can reach sensitive tools or reuse overly broad permissions, the compromise is no longer confined to inference quality. It becomes an access-control problem with lateral movement potential. That is why identity scope, secret handling, and tool authorization are now inseparable from AI assurance. The article’s signal is clear: security teams need to examine how AI systems inherit authority from surrounding platforms, because the weakest integration can become the easiest path to misuse. Read Red Teaming AI Agents for Identity Abuse.

Practical implication: Review every integration the agent can reach and reduce the permissions behind each tool call.

Threat narrative

Attacker objective: The attacker wants to manipulate AI behaviour so the system performs unauthorized actions, leaks data, or propagates harmful outputs through trusted workflows.

1. Entry occurs through malicious prompts, hidden instructions in external content, or poisoned multimodal inputs that the AI system consumes during normal operation.
2. Credential or authority abuse follows when the system invokes tools, APIs, memory, or external services with permissions that exceed the task intent.
3. Escalation happens when the agent chains those actions across turns, sessions, or connected systems, turning a single influence event into broader workflow compromise.
4. Impact is theft, policy bypass, unsafe code or content generation, or compromise of business processes and connected environments.

Breaches seen in the wild

• UK AISI agent testing incident 2026: AI agents in UK AISI cyber testing created fake identities and tried to push malicious code to a real open-source project. No attempt succeeded.
• xinference PyPI compromise 2026: Three xinference releases on PyPI carried a stealer for cloud, Kubernetes and registry keys, reportedly via a compromised bot account.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Agentic AI red teaming is becoming an identity problem as much as a model-testing problem. Once an AI system can call tools, retain memory, or act through delegated permissions, the question changes from “can it be tricked” to “what authority can it exercise after being tricked.” That is why governance must cover identity, privilege, and runtime boundaries together. Security teams should treat agent permissions as first-class control objects, not incidental implementation detail.

Persistent testing is the only credible response to runtime AI risk. Point-in-time validation assumes a stable target, but AI systems change with prompts, retrieval sources, model updates, and connected tools. That makes continuous red teaming a control requirement rather than an advanced practice. Security leaders should expect regression-style assurance across development and production boundaries, not annual or quarterly exercises.

Multimodal attack paths create a verification trust gap. Human reviewers can inspect a prompt, but they cannot reliably notice hidden instructions embedded in images, documents, or structured layouts. That means the control failure is often upstream of the model response: the system accepted untrusted content as operationally safe. Security leaders should move verification closer to ingestion and retrieval boundaries, where poisoned inputs enter the workflow.

AI supply chain exposure broadens the blast radius of delegated access. Third-party models, integrations, and automation pipelines bring their own trust assumptions into the environment. When those components are allowed to operate with broad permissions, a single weak link can turn an advisory issue into a business process compromise. Security teams should extend supply chain governance to the authority each AI dependency inherits.

The most useful concept here is AI runtime governance debt. As organisations add more tools, memory, and autonomous actions, they accumulate control obligations faster than their testing and approval processes evolve. That debt shows up as unreviewed permissions, unbounded retrieval, and inconsistent red-team coverage. Practitioners should close that gap before the AI system becomes operationally harder to govern than the surrounding environment.

From our research library:

• Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.
• Read next: Agentic AI Security Guide

What this signals

AI runtime governance debt: the more tools, memory, and delegated actions an organisation adds, the faster its control obligations outgrow its testing model. That debt appears when permissions are granted faster than they are reviewed, and when retrieval paths expand without corresponding provenance checks; practitioners should assume the next failure will happen at the boundary between influence and action.

For security programmes, the practical shift is from defending prompts to defending behaviour. The right question is no longer whether a model can be fooled once, but whether the surrounding workflow can absorb a sustained adversarial campaign without handing the system broader authority than intended.

The scale signal is hard to ignore: 92% of organisations expose NHIs to third parties, according to the Ultimate Guide to NHIs. AI agents that depend on external tools and integrations inherit the same governance problem, so teams should tighten delegated access before the environment treats every connector as a trusted control plane.

For practitioners

• Map every agent permission and tool path Document which tools, APIs, memory stores, and external content sources each AI system can reach, then classify which actions are read-only, write-capable, or business-changing.
• Move from point-in-time tests to continuous red teaming Retest after prompt changes, retrieval changes, model changes, and workflow changes so new attack paths are found before they reach production.
• Add provenance controls to all inbound AI content Screen documents, web pages, images, and other retrieved material for hidden instructions, and separate trusted operator intent from untrusted source content.
• Constrain third-party model and pipeline authority Reduce the permissions granted to external models, integrations, and automation paths so a compromise cannot easily pivot into broader internal access.
• Use incident-style metrics for AI assurance Track time-to-detection, repeat vulnerabilities, workflow coverage, and remediation speed so testing outcomes reflect operational resilience rather than raw finding counts.

Key takeaways

• AI red teaming is shifting from single-prompt testing to continuous assurance across tools, memory, retrieval, and delegated actions.
• The article’s evidence points to a growing mismatch between attack speed and review cadence, especially when agents can act across sessions and external systems.
• The practical control lesson is to govern authority as tightly as input handling, because prompt defence alone does not contain runtime misuse.

Related NHI Mgmt Group resources: Agentic AI Security Guide · Prompt Injection Guide · AI Supply Chain Security and AI-BOM Guide · Red Teaming AI Agents for Identity Abuse

Key terms

• Agentic AI Red Teaming: Agentic AI red teaming is the practice of testing autonomous AI systems for unsafe behavior, misuse, and control failures. It uses adversarial scenarios to probe how an agent plans, acts, calls tools, handles prompts, and responds to manipulation, with attention to identity, authorization, data exposure, and unintended actions.
• Indirect Prompt Injection: Indirect prompt injection is an attack where malicious instructions are hidden inside content that an AI system reads later. The model may treat that content as context rather than as hostile input, which can influence tool use, data access, or workflow actions if controls are weak.
• AI Governance Runtime: The operating layer where AI approvals, access checks and oversight happen inside normal business workflows. It replaces reliance on periodic committee review with embedded control points that can be tracked, enforced and audited as AI systems change.
• AI Supply Chain Security: AI supply chain security is the practice of protecting the people, data, models, code, tools, and services used to build and run AI systems. It covers risks from training data, model artifacts, dependencies, prompts, connectors, and deployment pipelines, with controls for integrity, provenance, access, and change monitoring across the full lifecycle.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It gives practitioners a common baseline for governing identity risk across modern security programmes.