Red teaming agentic AI workflows is now an execution-layer problem

At a glance

What this is: This is an analysis of red teaming agentic AI workflows in NVIDIA NeMo Agent Toolkit, and the key finding is that risk emerges across the full execution path rather than in the model alone.

Why it matters: It matters because IAM, NHI, and AI governance teams need controls that evaluate what an agent can access, call, and do across a workflow, not just what the base model says.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate.

👉 Read Lakera's research on red teaming agentic capabilities in NVIDIA NeMo Agent Toolkit

Context

Agentic AI changes the security question from what the model outputs to what the system can do during execution. When tools, memory, intermediate state, and handoffs are all part of the attack surface, traditional model-only testing misses the point because the failure often appears after the first decision, not before it.

For IAM and NHI teams, that means governance has to follow the workflow. The control problem is not just prompt safety or output filtering, but whether an agent can be evaluated across its real tool boundaries, delegated actions, and downstream effects. Lakera’s article uses the NeMo Agent Toolkit example to show why system-level red teaming is now part of AI access governance.

This is especially relevant as agentic systems move from experimentation into enterprise workflows where access, action, and accountability are tightly linked. The practical question is whether current control planes can observe and constrain the full execution path, or only the point where the model answers.

Key questions

Q: How should security teams red team agentic AI systems in practice?

A: Security teams should test agentic AI systems end to end, not just at the model prompt. Scenario-based red teaming should inject adversarial inputs at user, tool, and data-source boundaries, then measure whether the workflow propagates or contains the failure. The goal is to expose where risk originates, how it moves, and which controls actually change the outcome.

Q: Why do agent workflows create more governance risk than standalone models?

A: Agent workflows create more governance risk because they combine reasoning with action. Once tools, memory, and handoffs are involved, a failure in one step can affect later decisions or external actions even if the base model seems safe. Governance must therefore cover execution paths, not just outputs.

Q: What do organisations get wrong about AI agent risk scores?

A: They often treat risk scores as reporting rather than decision input. A useful score should change something concrete, such as access scope, tool restrictions, logging depth, or approval requirements. If the number does not lead to an operational change, it is not governance evidence.

Q: How do AI agents change identity governance for NHI programmes?

A: AI agents shift identity governance toward runtime authority. Instead of managing only static service accounts or API keys, teams must govern what the agent can do during execution, which tools it can call, and how far its decisions can propagate. That makes least privilege, auditability, and lifecycle control apply to behaviour as well as credentials.

Technical breakdown

Why agentic workflows create a larger attack surface than base models

Agentic systems do not fail only at the point of text generation. They fail across user input, tool invocation, external data sources, intermediate reasoning, memory, and handoffs between agents or steps. That matters because a prompt injection, unsafe tool response, or manipulated retrieval result can influence later actions even if the base model appears constrained. In other words, the workflow becomes the security boundary, not the model alone. Red teaming therefore has to test the path from input to action, not just the answer that comes out at the end.

Practical implication: evaluate agent workflows at each tool boundary and execution step, not only at the model output.

How structured red teaming exposes risk propagation in AI agent workflows

Structured red teaming works by defining a scenario, injecting an adversarial condition at a specific point, and measuring whether the agent reaches the attacker’s intended outcome. The useful part is not just whether the attack succeeds, but how risk spreads, amplifies, or is filtered across the workflow. Normalized scoring and attack success rate make runs comparable over time, while grouped views reveal whether failures cluster by scenario type or output condition. That gives teams evidence about whether a mitigation reduces the actual attack path or only shifts the failure elsewhere.

Practical implication: use repeatable scenario-based testing to compare risk before and after guardrails, prompt changes, or tool restrictions.

Why multi-step execution breaks conventional security assumptions

Conventional testing often assumes that if a component looks safe in isolation, the overall system is acceptable. Agentic workflows break that assumption because a safe-seeming component can still become dangerous once combined with other steps, tools, or handoffs. A direct request may fail, but an indirect input or unsafe downstream tool response can still trigger harmful behaviour later in the chain. This is why the article frames red teaming as system-level evaluation. The security question is whether the workflow remains resilient under adversarial conditions, not whether one component passes a standalone test.

Practical implication: treat isolated component testing as incomplete unless it is validated against end-to-end adversarial workflow behaviour.

Threat narrative

Attacker objective: The attacker’s objective is to cause the agent to produce harmful, misleading, or data-exposing behaviour through the workflow itself.

1. Entry occurs when an adversarial prompt or indirect input is injected into the agent workflow, including user input or a downstream data source.
2. Escalation happens when the injected condition influences tool calls, intermediate state, or multi-agent handoffs, causing the workflow to move beyond the original request.
3. Impact follows when the agent produces harmful output, leaks data, or takes an unsafe action that would not have occurred under benign execution.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Execution-layer governance is now the real control plane for agentic AI. Lakera’s framing is correct because the security boundary has moved from the model response to the workflow that surrounds it. Tool calls, memory, and handoffs create failure paths that a model-level check will never see in full. Practitioners should treat agent workflows as governed execution systems, not just smarter chat interfaces.

Workflow-level red teaming is the only way to see how agent risk propagates. The article’s strongest contribution is the distinction between component testing and end-to-end evaluation. A safe output at one step does not prove a safe system when adversarial input can influence later actions, tool responses, or multi-agent transitions. That means the security programme has to measure propagation, not just point failures.

AI agents are becoming non-human identities with broader privilege exposure than most teams admit. Once an agent can retrieve data, invoke tools, and act across enterprise workflows, its identity behaviour belongs in the same governance conversation as service accounts and workload identities. OWASP-NHI and ZT-NIST-207 become relevant because the issue is who or what can act, on what data, and with what downstream authority. Practitioners should reframe agent oversight as identity governance, not model governance alone.

Agentic risk demands a named concept: execution-layer exposure. The article shows that exposure is no longer limited to inputs or outputs. It exists in the sequence of tool use, state carryover, and delegated action that turns a model into an operational actor. That concept is useful because it separates surface-level prompt defence from the real control problem. Practitioners should map controls to execution paths, not generic AI risk categories.

Normalised scoring matters only if it is tied to governance decisions. Risk scores and attack success rates are useful because they make workflows comparable across releases, but only if teams use them to decide what gets blocked, monitored, or redesigned. Otherwise they become reporting artefacts with no control value. The governance test is whether a score change leads to a policy or architectural change.

From our research:

• 98% of companies plan to deploy even more AI agents within the next 12 months, despite documented rogue behaviour in 80% of current deployments, according to AI Agents: The New Attack Surface report.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
• OWASP Agentic AI Top 10 helps teams translate those runtime behaviours into concrete threat categories and control priorities.

What this signals

Execution-layer exposure: the next governance gap is not whether an agent can answer safely, but whether it can act safely across a chain of tools, data sources, and handoffs. As agent adoption scales, control design has to move from prompt moderation to workflow restraint, and the operational question becomes which steps should be observable, approval-gated, or denied outright.

With 52% of companies able to track and audit the data their AI agents access, the remaining 48% still lack a usable audit trail for incident response and compliance review. That is a programme-level problem, not a feature gap, because once agents can make decisions in motion, post-event explanation becomes much harder to reconstruct.

Teams that already manage service accounts and workload identities have a useful starting point, but agentic systems add non-deterministic action paths that static entitlement models were never built to absorb. The practical signal to watch is whether red team findings are changing access scope, tool restrictions, and approval logic, not just the wording of policies.

For practitioners

• Map agent workflows as governed execution paths Document every user input, tool call, intermediate state, and handoff that can change what the agent does. Treat those boundaries as access points and assess which ones need logging, approval, or denial.
• Red team the workflow, not only the model Run scenario-based tests that inject adversarial conditions at user input, indirect data sources, and tool boundaries. Compare attack success rate and normalized risk across releases so mitigation is measured against the full path.
• Tie agent permissions to the smallest usable task scope Limit which tools, data sources, and actions an agent can reach in each workflow stage. Separate read, write, and external action privileges so one compromise does not become a full workflow failure.
• Instrument risk propagation across steps Track where failures originate, where they amplify, and where they are filtered. That evidence helps distinguish a weak prompt boundary from a deeper tool or handoff problem that needs architectural change.

Key takeaways

• Agentic AI expands the attack surface from model output to full workflow execution, which makes system-level red teaming a governance requirement rather than an optional exercise.
• The strongest evidence comes from structured testing that measures attack success, risk propagation, and failure consistency across tool boundaries and handoffs.
• Identity teams should govern agentic systems as execution-capable non-human identities, with controls tied to task scope, observability, and delegated authority.

Key terms

• Agentic Workflow: An agentic workflow is a sequence of AI-driven steps that can retrieve data, call tools, carry state forward, and trigger downstream actions. In governance terms, the workflow is the security boundary because risk may emerge in transitions between steps, not only in the model response itself.
• Risk Propagation: Risk propagation is the way a failure in one part of a system influences later steps, outputs, or actions. For agentic systems, it describes how an unsafe input, tool response, or intermediate state can spread across the workflow and create a larger operational impact.
• Normalized Risk Score: A normalized risk score is a standardised measure used to compare how vulnerable an agent is across different scenarios or releases. It becomes useful only when teams use it to drive decisions about access, tool scope, guardrails, and monitoring.
• Attack Success Rate: Attack success rate is the percentage of test attempts in which an injected adversarial scenario achieved its intended effect. For agentic systems, it helps distinguish a one-off anomaly from a repeatable workflow weakness that requires governance action.

What's in the full article

Lakera's full research covers the operational detail this post intentionally leaves for the source:

• The sample Retail Agent configuration and how the evaluation point is set for workflow_output.
• Per-scenario attack success rate and normalized score output that lets teams compare failures across runs.
• The exact scenario structure used to test user input, indirect data sources, and tool boundaries.
• Examples of how score distributions reveal consistent versus intermittent agent failures.

👉 Lakera's full post covers the sample report, scenario metrics, and workflow evaluation details.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.