AI red teaming for agentic systems needs continuous testing

At a glance

What this is: This is Akto’s guide to AI red teaming for agentic systems, and its central finding is that AI security testing must be continuous because AI behaviour is probabilistic and attack paths change with context.

Why it matters: It matters to IAM, NHI, and AI security teams because agentic systems can inherit credentials, access tools, and act across workflows, which turns model testing into identity and access governance as well as AI assurance.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).
• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate.

👉 Read Akto's guide to continuous AI red teaming for agentic systems

Context

AI red teaming is a security testing method for systems that do not behave deterministically, which means the same input can produce different outputs depending on context, state, and prior prompts. That makes traditional penetration testing incomplete for large language models and agentic systems, because the real attack surface includes prompts, tool use, training data, inference pipelines, and integrated services.

The identity implication is direct: when an AI system can call tools, access data, or act on behalf of a user, it becomes a non-human identity governance problem as well as an AI security problem. Testing therefore has to examine not just model behaviour, but also what privileges the system can reach, how those privileges are constrained, and whether abuse can be detected before the agent completes its task.

A continuous testing model is now the baseline rather than a specialist exercise. That starting position is increasingly typical across organisations adopting agentic AI, especially where the system can interact with internal data, APIs, or operational workflows.

Key questions

Q: How should security teams run AI red teaming for GenAI systems?

A: Start with the system’s trust boundaries, then test prompts, retrieval sources, tool calls, and output handling together. Good AI red teaming does not stop at finding bad answers. It checks whether the system can be pushed into revealing data, bypassing policy, or taking unsafe actions through connected interfaces.

Q: Why do agentic AI systems require continuous red teaming?

A: Agentic systems change when models are retrained, prompts are updated, retrieval sources shift, or new integrations are added. Those changes can reopen vulnerabilities even if the model was previously tested. Continuous red teaming is how teams detect drift, validate fixes, and keep security evidence aligned to the current system state.

Q: What breaks when AI red teaming is treated like traditional penetration testing?

A: Teams miss the AI-specific attack surface, especially prompt handling, output manipulation, retrieval poisoning, and tool misuse. A standard pen test can prove infrastructure exposure, but it cannot reliably measure whether the model will follow malicious instructions or misuse delegated access. That leaves a major assurance gap for agentic systems.

Q: Who is accountable when an AI agent accesses sensitive data it was not meant to use?

A: Accountability sits with the team that approved the agent, its connectors, and its policy boundaries, not with the runtime behaviour alone. Organisations need ownership for intent, permissions, monitoring, and validation so they can prove whether the agent stayed inside its approved purpose. Without that, audit and regulatory response become retrospective guesswork.

Technical breakdown

Why AI red teaming differs from traditional penetration testing

Traditional penetration testing assumes a relatively stable target: the same input usually maps to the same result, and controls can be validated against known attack paths. AI red teaming works differently because model behaviour is probabilistic, context-sensitive, and often influenced by hidden system prompts, retrieval layers, and tool permissions. That means testers are probing both the model and the surrounding orchestration layer. The attack surface includes prompt handling, output conditioning, data sources, and delegation chains, not just exposed endpoints or software flaws.

Practical implication: test the AI system and its connected privileges together, not as separate assurance streams.

How prompt injection and tool abuse change the security model

Prompt injection turns untrusted text into an attack vector by persuading the model to ignore higher-level instructions or manipulate downstream tool use. In agentic systems, that becomes more serious because the model can take actions, not just generate text. Tool abuse happens when the system calls an API, writes data, or executes a workflow outside the intended policy boundary. This is where AI security intersects with NHI governance, because the agent may effectively act with service-account-like privileges without being managed as an identity.

Practical implication: bind tool permissions to narrowly scoped, auditable policy and treat agent actions as governed identity events.

Why continuous validation is the only realistic control model

AI red teaming has to be iterative because model updates, retrieval changes, new integrations, and fine-tuning can reopen vulnerabilities that were previously closed. A single assessment only captures one state of the system. Continuous validation uses manual discovery to find novel attack patterns, then automation to rerun those patterns as regression tests. That approach is especially important for agentic AI because behaviour can drift without code changes, and an apparently safe model can become risky once connected to new tools or data sources.

Practical implication: build red teaming into release and change management so regressions are detected before production exposure.

Threat narrative

Attacker objective: The attacker wants to make the AI system behave outside policy so that it leaks information, executes unauthorised actions, or becomes a bridge into connected systems.

1. Entry occurs when an attacker uses prompt injection, malicious instructions in retrieved content, or manipulated user input to influence the AI system.
2. Escalation follows when the model accepts those instructions and abuses connected tools, data access, or workflow permissions beyond the intended scope.
3. Impact occurs when the agent exposes sensitive data, performs unauthorised actions, or corrupts downstream decisions in a live business process.

NHI Mgmt Group analysis

Continuous AI red teaming is now an identity governance control, not only a model safety exercise. Once an AI system can call tools or touch internal data, it behaves like a governed non-human identity with delegated access. That shifts the question from whether the model is safe in the abstract to whether its action boundary is enforceable, reviewable, and revocable. Practitioners should treat red teaming as part of access governance for agentic systems.

Tool access is the real blast-radius driver in agentic AI. The model prompt matters, but the larger risk is what the system can do after a successful manipulation. If an agent can read, write, call, or trigger workflows with broad privileges, prompt injection becomes an access-control failure rather than a purely linguistic one. The operational conclusion is that least privilege must extend to AI orchestration layers and every connected secret or token.

Probabilistic behaviour creates governance debt when teams rely on one-time assurance. A model that passed yesterday’s test can fail after retraining, retrieval changes, or a new connector. That makes static sign-off weak evidence for a live agentic estate. Security leaders should expect red teaming to become a standing control with measurable regression testing, not a periodic project.

AI red teaming needs named accountability across security, legal, and compliance. The article correctly points to governance as the mechanism that turns findings into action, but the larger point is that AI incidents do not stay inside the model team. Ownership must cover data access, workflow permissioning, audit evidence, and incident response. Practitioners should anchor accountability before agents reach production.

What this signals

Agentic AI testing will converge with privileged access governance. The operational boundary is no longer just the model, it is the set of tools, secrets, and APIs the system can reach. That means AI security teams and IAM teams will need common evidence for permissions, auditability, and revocation. The practical shift is toward continuous control validation across both model behaviour and delegated access.

AI governance debt accumulates faster than most release processes can absorb. Every new connector, retrieval source, or fine-tune adds attack paths that are easy to overlook in a one-time approval flow. In practice, the strongest teams will build red teaming into change management, not treat it as a separate specialist review. That is the only way to keep assurance current as agent behaviour drifts.

Prompt injection is becoming a proxy for access failure. As agentic systems absorb more operational tasks, a successful prompt manipulation can produce the same outcome as an overprivileged service account or a mis-scoped API token. The control objective is therefore not only content moderation but limiting what the agent can reach, change, or expose under pressure.

For practitioners

• Scope the AI system as a governed identity surface Document every model, connector, retrieval source, API token, and workflow the agent can reach. Map which actions are read-only, which can mutate data, and which can trigger external effects so testing reflects actual blast radius.
• Run manual red teaming before automating regression tests Use human testers to discover novel prompt injection, jailbreak, and tool-abuse patterns first, then convert the highest-value cases into repeatable test suites for release gates and production monitoring.
• Treat agent permissions like privileged access Apply least privilege to tool calling, data retrieval, and delegated actions. Separate read, write, and execute scopes, and rotate or revoke any secret that gives an agent broader reach than the task requires.
• Record prompts, model state, and environment conditions Capture the exact prompt, system version, retrieval set, and test environment whenever a finding reproduces. Without that evidence, teams cannot validate fixes or prove that a regression was contained.
• Tie red team findings to governance and compliance workflows Escalate validated failures into risk registers, access reviews, and incident response playbooks so legal, compliance, and security owners can track remediation and sign-off decisions together.

Key takeaways

• AI red teaming is moving from a niche testing discipline to a standing control for agentic systems.
• The biggest risk is not just model misbehaviour, but the access the model can reach once manipulated.
• Teams that combine human discovery with automated regression testing will have the best chance of keeping pace with changing AI attack paths.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Tool Abuse: Tool abuse occurs when an AI assistant is manipulated into making actions through connected APIs, plugins, or workflows that the user did not intend. The risk is not only incorrect output, but execution of legitimate systems with attacker-chosen inputs or destinations.
• Agentic AI: Autonomous AI systems capable of planning, deciding, and taking actions — including calling APIs, writing code, and orchestrating other agents — with minimal human oversight. Agentic AI introduces new NHI risks as agents must authenticate to external services.

What's in the full article

Akto's full blog covers the operational detail this post intentionally leaves for the source:

• Step-by-step scoping of AI red team engagements across prompts, integrations, data sources, and lifecycle stages.
• Examples of manual versus automated attack patterns, including discovery, escalation, persistence, and validation workflows.
• Guidance on tracking prompts, system state, and remediation evidence across iterative model changes.
• Practical discussion of governance, legal, and compliance coordination for AI red team findings.

👉 Akto's full post covers the red teaming process, test categories, and governance considerations in more operational detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps practitioners translate identity controls into operational guardrails for AI-enabled environments.