TL;DR: AI red teaming must evaluate entire agentic systems, not just isolated model endpoints, because tool use, APIs, data retrieval and workflow steps expand the attack surface, according to DeepKeep research. The shift makes reproducible replay, deterministic evidence and remediation-ready outputs central to enterprise AI assurance.
At a glance
What this is: This is an analysis of OWASP-aligned AI red teaming that says effective testing has to cover the full agentic workflow, including tool calls, APIs, data access and replayable evidence.
Why it matters: It matters because IAM, NHI and AI security teams need to govern what agents can do at runtime, not just what the model says in isolation.
Context
AI red teaming is no longer just prompt injection testing against a model endpoint. As agentic systems gain access to tools, APIs, databases and sensitive data, the security question shifts to whether the whole workflow can be abused, not whether a single output can be manipulated.
That creates a governance gap for both AI and identity teams: the control boundary now includes delegated actions, retrievable context and runtime privilege. AI red teaming guide and similar practices matter because testing needs to follow the system path that an attacker would actually exploit.
Key questions
Q: How should security teams test GenAI systems for prompt injection?
A: Test the full path, not just the chat box. Include direct prompt injection, indirect injection through documents and webpages, memory abuse, encoded text, and image-based prompts. The goal is to see whether the model can be pushed into unsafe tool use or disclosure when inputs look trusted but are not.
Q: Why do agentic AI incidents quickly become identity and privilege problems?
A: Because once an agent reaches credentials or inherited trust, the incident is no longer about model output. The risk shifts to how far the agent can move with those permissions, which systems it can touch, and whether those privileges were ever meant to be reusable across tasks or environments.
Q: What are the signs that AI red teaming is too shallow?
A: The usual signs are narrow test cases, no coverage of live workflows, no retesting after model or prompt changes, and findings that are not linked to remediation. If the programme only checks model accuracy or one-off demo prompts, it is not exercising the actual attack surface the business depends on.
Q: Should organisations separate model safety testing from workflow security testing?
A: No. Model safety and workflow security overlap, but they are not the same problem. A model can behave acceptably in isolation while the integrated system still permits unauthorized retrieval, tool abuse or privileged action. Organisations should evaluate both, but the workflow test should govern real deployment decisions.
Technical breakdown
Why isolated model testing misses agentic risk
An LLM can appear safe in a narrow chat test while the surrounding agentic workflow remains exploitable. The real attack surface includes tool invocation, retrieval pipelines, API calls, database access and intermediate state. When the evaluation stops at the model endpoint, it misses the path an attacker would use to turn a malformed prompt into an unauthorized action. That is why workflow-level testing is more useful than static prompt lists: the threat is not just incorrect text generation, but harmful orchestration across connected systems.
Practical implication: Test the full execution path, not just the model response, before treating an AI system as production-ready.
How deterministic replay changes AI remediation
Deterministic replay matters because AI failures are often multi-step and stateful. If you cannot reproduce the exact prompt, tool call, retrieval result and agent decision chain, engineering teams cannot prove whether a fix actually closed the issue. Replay also makes guardrail tuning and prompt adjustments auditable, which is essential when the same failure can emerge through slightly different contexts. In practice, replay turns AI red teaming from a one-off assessment into a repeatable control-validation process.
Practical implication: Require logged attack traces and replayable sessions so remediation can be verified, not assumed.
Why identity and privilege are now part of AI security testing
Agentic AI changes the security model because the system can act through delegated permissions rather than only by generating content. That makes privilege scope, tool authorization and data access central testing concerns. A red team therefore has to ask whether the agent can exceed its intended authority through external APIs, chained actions or overly broad retrieval rights. This is where AI governance overlaps with identity governance: if the agent can act, it effectively needs an identity boundary and a privilege model. Read Red Teaming AI Agents for Identity Abuse.
Practical implication: Treat agent permissions and tool entitlements as testable security controls, not implementation details.
Threat narrative
Attacker objective: The objective is to make the agent perform unauthorized actions or disclose sensitive data through the permissions and tools already available to it.
- Entry occurs when an attacker crafts a context-aware prompt or multi-turn interaction that targets the agentic workflow rather than the model alone.
- Credential_harvested or privileged access is abused when the agent is induced to invoke tools, query databases or call external APIs beyond intended scope.
- Escalation follows when the workflow allows unauthorized retrieval, broader data access or privilege escalation through connected systems.
- Impact is the unauthorized action, data exposure or operational misuse that results from the agent being steered through its own delegated capabilities.
Breaches seen in the wild
- tj-actions/changed-files compromise 2025: A stolen bot token let attackers poison tj-actions/changed-files so pipelines printed their CI/CD secrets to public logs (CVE-2025-30066).
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
OWASP-aligned AI red teaming has to move from content safety to delegation safety. The article is really about the collapse of a common testing assumption: that model behaviour can be judged separately from the tools it can use. Once an AI system can call APIs, query databases or trigger workflows, the question becomes whether its delegated actions are bounded. Practitioners should treat tool access as part of the security test, not an implementation footnote.
Deterministic replay is the difference between finding a weakness and governing it. Multi-step agent failures are often impossible to validate without step-by-step logs that capture prompts, responses, retrievals and tool outputs. That makes replay an assurance control, not just an engineering convenience. If teams cannot reproduce the abuse path, they cannot prove the fix worked, and that weakens both AI governance and incident response readiness.
Agentic workflows create a new trust boundary that identity teams must own. When an agent can retrieve data and invoke tools, its privileges behave more like a non-human identity than like a static application feature. AI agent identity abuse is therefore not a niche concern but a core governance problem for AI operations. Practitioners should design testing around the privileges the agent actually holds, not the prompts it happens to receive.
DeepKeep's article sharpens a useful concept: workflow-level AI exposure. The risk is not simply that an LLM can be jailbroken, but that an integrated agent can be manipulated into taking sanctioned-looking actions across connected systems. That changes the control objective from output moderation to runtime containment. Teams should govern the workflow path as aggressively as the model itself.
The market signal is clear: AI red teaming is becoming a control validation discipline, not a demo exercise. Enterprises do not need more static jailbreak libraries. They need repeatable testing that measures whether agent behaviour stays within authorised boundaries as context, data and tooling change. Practitioners should expect assurance programmes to shift toward evidence-rich replay, runtime privilege checks and remediation verification.
Read next: AI Security Maturity Model
What this signals
Workflow-level AI exposure: enterprise AI governance is moving toward control validation at the agent boundary, where permissions, tools and retrieval paths converge. That means assurance programmes need evidence that the system stays inside its authorised operating envelope when context changes, not just when prompts are clean.
The practical shift for security teams is toward continuous verification. As agent behaviour becomes more dynamic, static policy documents will matter less than reproducible tests that prove tool use, data access and escalation paths remain bounded over time.
For practitioners
- Define the full agentic attack surface Map every tool, API, database and retrieval path the agent can reach so red teaming covers the entire workflow instead of the model endpoint alone. See the AI Red Teaming Guide.
- Log every step of each test run Capture prompts, responses, intermediate decisions and tool outputs so successful abuse paths can be replayed deterministically during remediation.
- Review delegated privileges before testing Inventory the permissions granted to agents and reduce overbroad access before you start assessing how far a malicious prompt can push them.
- Use context-aware adversarial scenarios Replace static jailbreak lists with multi-turn scenarios that incorporate the application's own context, seed data and expected workflow behaviour.
- Treat replay evidence as a release gate Require a verified replay of the abuse chain before closing findings so fixes are proven against the exact sequence that triggered the issue.
Key takeaways
- AI red teaming now has to evaluate what an agent can do across tools, APIs and workflows, not just what a model says in isolation.
- The strongest evidence comes from deterministic replay, because multi-step agent failures are otherwise hard to reproduce and verify.
- Identity and privilege are part of the AI testing problem whenever an agent can act through delegated permissions or broad data access.
Related NHI Mgmt Group resources: Red Teaming AI Agents for Identity Abuse · AI Red Teaming Guide · AI Security Maturity Model · AI Governance Platform Buyer's Guide
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI02 — Tool Misuse | The article focuses on agents abusing tools, APIs and workflow actions during red teaming. |
| ASI03 — Identity & Privilege Abuse | Delegated permissions and overbroad access are central to the article's AI governance concern. | |
| Recommendation — Test agent tool use for abuse paths and block any action the agent cannot justify for the task. Review agent privileges and reduce any access that is not required for the intended workflow. | ||
| NIST AI RMF | GOVERN — AI Governance and Accountability | The article is about governing how AI systems are assessed and controlled in enterprise settings. |
| MANAGE — Risk Management | Workflow-level red teaming is a risk treatment activity for agentic systems. | |
| Recommendation — Assign clear accountability for agent testing, evidence retention and remediation approval. Use test findings to prioritise containment of the highest-risk agent actions and access paths. | ||
| MITRE ATLAS | TA0006; TA0008 — Credential Access; Lateral Movement | The abuse chain can include credential-like access to tools and movement across connected systems. |
| Recommendation — Map agent abuse paths to credential access and lateral movement so detections cover multi-step attacks. | ||
Key terms
- Agentic Workflow Authorization: Agentic workflow authorization is the control approach used when an AI agent can select tools and continue acting across multiple steps. The key requirement is to evaluate each action in context, because one approval rarely covers the full sequence of decisions an autonomous actor can take.
- Deterministic Replay: Deterministic replay means a security decision can be reproduced from the same inputs and context without changing the outcome. In AI-assisted analysis, it is the property that makes cached answers trustworthy because the model interaction, evidence, and instructions are preserved exactly.
- Delegated AI Privilege: Delegated AI Privilege is the authority an AI agent receives from a human, application, or system to perform specific actions on its behalf. In identity security, this privilege must be bounded by policy, scoped to least privilege, time-limited, and auditable so agent actions remain attributable, controlled, and revocable.
Deepen your knowledge
NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance and workload identity alongside the identity controls that matter in modern security programmes. It helps practitioners connect identity boundaries to the broader assurance work their teams already manage.
Published by the NHIMG editorial team on October 22, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org