AI red teaming for agentic systems needs continuous testing

At a glance

What this is: This is a guide to AI red teaming for agentic systems, with the central finding that continuous adversarial testing is needed because AI behaviour changes with context, prompts, and system state.

Why it matters: It matters because agentic AI creates security failure modes that conventional penetration testing and periodic assurance cannot reliably expose, especially where prompts, tools, and connected data sources can be manipulated.

By the numbers:

• Attacks with roleplay cost 89.6% success, according to Akto.

Context

Agentic AI changes the testing problem because the system is not deterministic in the way conventional software is. The same prompt can produce different outputs depending on context, model state, connected tools, and surrounding data, so a single red-team pass cannot establish durable assurance. For identity and access teams, that makes AI agent behaviour part of the governance surface, not just a model-quality issue.

The security gap is not only in the model, but also in the surrounding control plane. Prompt handling, tool invocation, training data integrity, and system integrations all create places where an attacker can influence decisions or extract information. That is why AI red teaming for agentic systems has to be continuous, lifecycle-based, and tied to the way the system is actually deployed.

Key questions

Q: Why do agentic AI systems require continuous red teaming?

A: Agentic systems change when models are retrained, prompts are updated, retrieval sources shift, or new integrations are added. Those changes can reopen vulnerabilities even if the model was previously tested. Continuous red teaming is how teams detect drift, validate fixes, and keep security evidence aligned to the current system state.

Q: Why do agentic AI systems need continuous testing instead of periodic pen tests?

A: Agentic systems are probabilistic, so the same input may behave differently across sessions, model states, and tool contexts. Periodic pen tests miss those changes. Continuous testing matters because assurance must track the evolving trust boundary, not just a point-in-time configuration.

Q: What do security teams get wrong about prompt injection in agentic workflows?

A: They often treat prompt injection as an input validation problem only. In reality, the risk appears when hostile context is converted into action, especially if the agent can run commands, call tools, or make network requests. The control objective is to prevent unsafe action, not just detect bad text.

Q: What happens when an agentic system is tested only once before release?

A: A one-time test can give false confidence because the model, prompts, tools, and connected data sources keep changing after launch. A control that worked in one session may fail in another. That leaves organisations exposed to regressions, repeated jailbreaks, and tool abuse that no longer match the original test conditions.

Technical breakdown

Why probabilistic AI behaviour breaks one-time testing

Traditional security testing assumes repeatability: the same input should expose the same behaviour each time. Agentic AI does not work that way because context, prior turns, tool outputs, and model updates all influence the next action. A prompt that fails in one run may succeed in another, especially where hidden instructions, jailbreak patterns, or indirect prompt injection are involved. That means the testing objective is not just finding a flaw once, but proving whether the flaw remains exploitable across states and versions. Practical implication: build tests that are repeatable enough to measure drift, not just exploitability.

Practical implication: Run adversarial tests against multiple sessions, model versions, and environment states so you can see whether a weakness is stable or transient.

How prompt injection, poisoning, and tool abuse change the attack surface

Agentic systems expand the attack surface beyond the model prompt itself. Prompt injection can redirect behaviour, data poisoning can corrupt what the system learns or retrieves, and tool abuse can turn an apparently benign request into unauthorised actions through connected APIs or workflows. These are not abstract AI risks. They are control failures around instruction hierarchy, input trust, and delegation boundaries. In an agentic environment, the question is not only whether the model can be tricked, but whether the surrounding tools will accept the trick as valid intent. Practical implication: test the full chain from user input to external action, not just the response text.

Practical implication: Validate instruction boundaries, retrieval sources, and tool permissions as one attack path so you can see where manipulation becomes execution.

Why hybrid red teaming is the right operating model

Manual and automated testing solve different problems. Human testers are better at discovering novel attack paths, roleplay-based jailbreaks, and subtle logic traps. Automated testing is better at scale, regression coverage, and repeated validation after model or integration changes. For agentic systems, the hybrid model is the only one that gives both discovery depth and operational continuity. The test cases that matter most should be converted from proven manual findings into automated regression checks. Practical implication: treat red-team findings as living test assets, not as one-off reports archived after release.

Practical implication: Use manual probing to find new failure modes, then automate the proven attacks so every model change is re-tested.

Threat narrative

Attacker objective: The attacker wants to hijack agent behaviour so the system reveals data, executes unintended actions, or accepts unsafe instructions as legitimate.

1. Entry begins when an attacker uses prompt manipulation, indirect instructions, or poisoned content to influence the agent's interpretation of a task.
2. Credential or tool abuse follows when the agent treats the malicious instruction as legitimate and invokes connected systems, retrieves sensitive data, or performs unauthorised actions.
3. Impact occurs when the manipulated agent leaks information, executes harmful workflows, or persists in an unsafe behaviour pattern across repeated runs.

Breaches seen in the wild

• SalesBleed Salesforce Agentforce 2026: Three fixed Agentforce flaws let poisoned web leads make AI agents leak CRM data with zero clicks and send phishing under the agent's identity.
• Meta Muse agent hijack 2026: An undocumented Muse setting let local malware hijack Meta's personal AI agent, steal its authentication material and abuse user access.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Continuous testing is now the correct control model for agentic AI. A one-time red team cannot keep pace with probabilistic behaviour, model updates, and changing tool chains. The article makes the underlying point clearly: if the system can change its response from one run to the next, assurance has to move with it. For practitioners, the operational question is not whether red teaming exists, but whether it is embedded into the AI lifecycle.

Agentic AI creates an identity and delegation problem, not just a model-risk problem. Once an AI system can call tools, access data, or act through integrations, the security question shifts to who or what is being authorised. That makes prompt handling, tool permissions, and session boundaries part of the identity control plane. Teams that treat the agent as just another application will miss the governance layer that actually determines blast radius.

Prompt manipulation is the clearest named failure mode here. The article shows that basic prompt attacks can produce real compromise, which means the control gap is not theoretical guardrail weakness but an acceptance boundary that is too permissive. That same pattern also intersects with OWASP-AGENTIC ASI03 Identity & Privilege Abuse and ASI02 Tool Misuse, because the weakness appears when the system accepts untrusted instructions as authority. Practitioners should treat this as a delegation failure first and a model failure second.

Red-team evidence should feed governance, not remain a security artefact. The article repeatedly points to documentation, validation, and continuous oversight, which is the right direction for NIST-AIRMF GOVERN and MEASURE functions. For agentic AI programmes, the point is to make risk visible in governance decisions, model release gates, and change management. The practical conclusion is simple: if red-team findings do not change release criteria, they are not being operationalised.

Agentic trust boundary drift: as tools, prompts, and retrieval sources change, the line between legitimate instruction and hostile manipulation moves too. That drift is what makes static assurance fail. The practitioner response is to define and re-test trust boundaries continuously, because the boundary itself becomes part of the attack surface.

From our research library:

• Gartner predicts that by the end of 2026, 40% of enterprise apps will feature task-specific AI agents.
• Read next: Agentic AI Security Guide

What this signals

Agentic trust boundary drift: the most important governance problem is not a single failed test, but the way prompts, tools, and retrieval sources keep moving the line between legitimate instruction and hostile manipulation. Security teams need to re-test that boundary whenever the model, the prompt stack, or the integration layer changes.

Continuous validation is becoming the practical control for agentic systems because the attack surface changes with context and tool access. That means red-team output should feed release criteria, monitoring, and incident playbooks, not sit in a separate assurance report.

The scale of adoption pressure is already visible: 40% of enterprise apps will feature task-specific AI agents by the end of 2026, according to the 2026 Infrastructure Identity Survey. That makes test repeatability and governance traceability the next operational baseline, not optional maturity work.

For practitioners

• Shift to continuous AI red teaming Retest agentic systems after model updates, prompt changes, new tools, and integration changes so assurance follows the lifecycle rather than the release date.
• Separate manual discovery from automated regression Use human testers to find novel jailbreaks, roleplay attacks, and logic traps, then convert validated cases into automated checks that run on every build and deployment.
• Test the full prompt-to-tool chain Trace how untrusted input moves from user text into retrieval, tool calls, and external actions, and verify that each handoff enforces its own trust boundary.
• Bind red-team findings to release gates Require documented remediation or risk acceptance before agentic systems can ship, and track whether the same exploit still succeeds after mitigation.

Key takeaways

• Agentic AI testing fails when teams treat model behaviour as stable, because context, tools, and prompts can change the outcome of the same input.
• The most relevant attack paths are prompt injection, data poisoning, model extraction, and tool abuse, and those paths become more serious once the agent can act beyond text generation.
• Continuous red teaming matters because mitigation has to be proven repeatedly, not assumed from a single assessment.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Tool Abuse: Tool abuse occurs when an AI assistant is manipulated into making actions through connected APIs, plugins, or workflows that the user did not intend. The risk is not only incorrect output, but execution of legitimate systems with attacker-chosen inputs or destinations.
• Agentic trust boundary: The point at which an AI system stops being merely authenticated and becomes operationally authorised to take its next action. In agentic environments, this boundary can move during execution, so governance must define and monitor it as a runtime control rather than a one-time enrollment decision.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is designed for practitioners who need to connect access, trust boundaries, and governance across modern identity programmes.