AI red teaming for agentic systems needs continuous validation

At a glance

What this is: This is an analysis of AI red teaming for LLMs and agentic workflows, with the key finding that periodic testing is inadequate for probabilistic, tool-using systems.

Why it matters: It matters because IAM and security teams must govern prompts, tools, and delegated actions as part of the access model, not as an afterthought to application testing.

👉 Read Akto's analysis of AI red teaming best practices for agentic AI security

Context

AI red teaming is the practice of attacking AI systems to see how they fail under adversarial pressure. In agentic environments, that means testing prompts, memory, tools, and delegated actions together because the security problem is no longer only model output quality, but runtime control of what the system can do.

Traditional penetration testing assumes relatively stable behaviour and clear exploit boundaries. Agentic AI breaks that assumption because the same prompt can produce different outcomes across runs, tools expand the attack surface, and trust can cross between agents or external content. That is why this topic sits at the intersection of AI governance and identity governance, especially where AI systems hold credentials or act on behalf of users.

Key questions

Q: What breaks when AI security is handled only at launch time?

A: Controls go stale as the system changes. New data sources, fine-tuning, prompt updates, and tool integrations can all expand the attack surface after go-live, which means the original approval no longer reflects the current risk. Continuous review is required because AI systems are operationally dynamic, not static software artifacts.

Q: Why do agentic AI systems complicate existing IAM and PAM controls?

A: They complicate them because IAM and PAM were built around stable identities, human-paced approvals, and entitlements that are reviewed after use. Agentic systems compress decision-making into runtime, which means access can be consumed, combined, and discarded before a review cycle ever sees it. That makes static privilege models incomplete for autonomous execution.

Q: How do security teams know if AI red teaming is working?

A: AI red teaming is working when testing finds real prompt injection paths, over-scoped integrations, and policy gaps before attackers do, and when fixes are re-tested successfully after model or workflow changes. The strongest signal is repeatable reduction in exposed authority, not a lower number of red-team findings on its own.

Q: What should teams do when an agent can access sensitive tools or data?

A: They should reduce scope before production use, separate high-risk tools from low-risk ones, and require explicit approval for any action with external impact. If the workflow cannot be safely constrained, the access model is too permissive for an agentic system.

Technical breakdown

Why AI red teaming differs from traditional penetration testing

Traditional penetration testing targets deterministic systems, where a known payload usually produces a predictable outcome. AI systems are probabilistic, so the same input can be safe in one run and harmful in the next. That makes point-in-time validation weak for agentic workflows, where prompts, memory, tools, and external context all shape behaviour. Red teaming therefore has to test outcomes under variation, not just one exploit path. The objective is to measure how the system behaves when instructions conflict, context shifts, or tool outputs are adversarial.

Practical implication: Treat AI security testing as a continuous control, not a launch gate.

Prompt injection, jailbreaks, and tool misuse in agentic workflows

Prompt injection is the most direct way to override intended model behaviour by embedding malicious instructions in user input or external content. Jailbreaking aims to bypass policy controls altogether, while tool misuse occurs when an agent is manipulated into using legitimate capabilities in harmful ways, such as querying systems it should not or executing untrusted code. The risk increases when agents connect to APIs, file systems, or other services because those tools turn a language model mistake into an operational action. In identity terms, the agent’s authority becomes part of the threat surface.

Practical implication: Bound tool permissions before testing, because every exposed capability expands blast radius.

Continuous validation and regression testing for AI security posture

AI security posture management needs runtime validation because models, prompts, and toolchains change constantly. A fix that blocks one attack pattern can fail after a prompt update, connector change, or new agent workflow. Effective programmes convert each discovered bypass into a regression test and run those tests automatically in CI/CD and at runtime. That creates a feedback loop between adversarial discovery and operational enforcement. For governed environments, the value is evidence: teams can prove that controls still hold after change, not merely that they once worked.

Practical implication: Build every high-severity finding into an automated regression test and rerun it after each change.

Threat narrative

Attacker objective: The attacker wants to steer an AI system into unsafe tool use, policy bypass, or unauthorized action that has real operational consequences.

1. Entry occurs through prompt injection, indirect prompt injection, or adversarial content that the model treats as trusted context.
2. Escalation happens when the agent follows malicious instructions into tool use, data access, or cross-agent trust decisions beyond its intended scope.
3. Impact is achieved when the agent exposes sensitive data, executes unsafe actions, or corrupts downstream decisions in a production workflow.

NHI Mgmt Group analysis

AI red teaming is becoming a control assurance function, not a specialist exercise. Once AI systems can call tools and trigger downstream actions, the question is no longer whether a model generates bad text. The question is whether the system can be manipulated into taking harmful actions inside a live environment. That shifts red teaming from a periodic assessment into a governance mechanism for runtime behaviour, especially where the agent has access to credentials, connectors, or business systems. Practitioners should align testing with AI RMF GOVERN and MEASURE outcomes, not just model quality checks.

Tool authority is the real attack surface in agentic AI. A model that can only speak is limited; a model that can query systems, execute code, or move data inherits the privileges of those integrations. That is where identity governance becomes relevant, because agent access should be treated as delegated authority with explicit scope, monitoring, and revocation. Without that discipline, the agent is effectively a non-human identity with oversized standing privilege. Security teams should design controls around what the agent can do, not only what it is allowed to say.

Continuous adversarial testing exposes AI governance debt. Many organisations have begun piloting agentic systems faster than they have built the control plane to monitor them. That creates a gap between deployment speed and evidence quality, which is exactly where hidden failures accumulate. AI governance debt: the growing gap between how quickly AI systems are deployed and how slowly their access, tool, and oversight controls are validated. Practitioners should treat each new tool connector or workflow change as a governance event that requires renewed testing.

Agentic AI security now overlaps with non-human identity governance. If an AI system can authenticate, retrieve data, and invoke tools, then it needs lifecycle controls similar to other high-risk NHI classes. That includes least-privilege scope, auditable entitlements, and a clear offboarding path when tools or workflows change. The discipline is no longer just model safety. It is identity-controlled access to action. Practitioners should bring IAM and PAM thinking into AI security reviews early, before agent sprawl creates opaque privilege paths.

Framework-led testing is the only way to make findings repeatable. Adversarial AI testing produces noise unless it is mapped to a consistent threat model. OWASP Agentic AI Top 10, MITRE ATLAS, and NIST AI RMF give teams a vocabulary for classifying prompt injection, tool misuse, and governance gaps in a way that survives beyond a single red team exercise. The practical outcome is better prioritisation. Teams should use frameworks to turn isolated findings into control coverage and board-readable evidence.

What this signals

AI governance debt will become a practical programme risk wherever teams deploy agents faster than they can monitor tool authority. The control gap is not abstract. It shows up when prompts, connectors, and memory are changed without renewed validation, and the safest response is to make those changes trigger a governed review cycle backed by evidence from red teaming and runtime telemetry.

For identity teams, the next shift is to treat agent access as a lifecycle problem. That means onboarding, scoped entitlements, monitoring, and offboarding for AI systems that can authenticate or act on behalf of users. The more an agent can do, the more it resembles a non-human identity that needs structured governance rather than informal oversight.

Security programmes should also expect red team evidence to become a governance artefact. When findings are tied to frameworks such as the OWASP Agentic AI Top 10 or MITRE ATLAS adversarial AI threat matrix, they become easier to prioritise, audit, and defend in risk committees.

For practitioners

• Map agent authority before testing begins Document every prompt source, memory store, tool connector, and external API the agent can reach. Treat each integration as a trust boundary and define the minimum scope needed for the workflow, especially where the agent can act on data or credentials.
• Convert every exploit into a regression case Capture successful prompt injections, jailbreaks, and tool misuse chains as automated tests. Re-run them after model updates, prompt changes, connector additions, and workflow changes so previously blocked paths do not reopen silently.
• Restrict agent tool permissions to task scope Remove unnecessary access to code execution, file systems, and external systems before production deployment. Apply the same review discipline you would use for a privileged service account, because agent tools are effectively delegated authority.
• Add runtime monitoring for policy bypass signals Watch for repeated prompt variation, abnormal tool calls, unauthorized data retrieval, and cross-agent contamination. Feed those signals into security operations and governance reporting so red team findings become operational evidence, not isolated defects.

Key takeaways

• Agentic AI red teaming is a runtime control discipline because static assessment cannot keep pace with changing prompts, tools, and workflows.
• The central risk is delegated authority, where an AI system behaves like a non-human identity with tool access that can be manipulated into unsafe actions.
• Mature programmes turn every discovered bypass into a regression test and use frameworks to prove that controls still hold after change.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Tool Misuse: Tool misuse occurs when an agent uses an allowed integration in a way that exceeds its intended task, scope, or risk tolerance. The problem is often not access alone but the combination of valid credentials, broad permissions, and unbounded action sequencing.
• AI Governance: AI governance is the set of controls used to discover, classify, approve, restrict, monitor, and revoke AI-enabled access. It connects identity, data, and policy so organisations can manage what AI can reach, what it can share, and when it should be stopped.

What's in the full article

Akto's full blog covers the operational detail this post intentionally leaves for the source:

• Step-by-step red teaming workflow for planning, testing, reporting, and retesting AI systems.
• Practical examples of prompt fuzzing, jailbreak attempts, and tool abuse simulations across agentic workflows.
• Guidance for building continuous AI security posture management into CI/CD and runtime operations.
• Framework mapping details for OWASP, MITRE ATLAS, and NIST AI RMF in AI red team programmes.

👉 Akto's full post covers prompt injection, tool misuse, runtime validation, and continuous red team workflows.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and machine identity security. It helps practitioners connect identity controls to emerging AI and automation risks across their programmes.