AI red teaming now spans the full AI stack and agentic risk

At a glance

What this is: This is TrojAI’s analysis of how AI red teaming has expanded from model-level testing to full-stack adversarial simulation, with agentic workflows now central to the risk picture.

Why it matters: It matters because IAM, NHI, and security architecture teams now have to govern dynamic AI behaviour across tools, data, and runtime execution, not just authenticate a model or user.

👉 Read TROJ.AI's analysis of AI red teaming across the full AI stack

Context

AI red teaming is the practice of trying to break an AI system before attackers do. In this article, the primary issue is no longer model accuracy alone but the integrity of the full AI stack, including retrieval, tools, infrastructure, and deployment paths.

For identity and access programmes, that shift matters because AI systems increasingly act through non-human credentials and external integrations. Once an AI workflow can call tools, touch data, and trigger actions, governance moves from model testing into identity control, privilege scope, and runtime oversight.

Key questions

Q: How should security teams test AI systems that can use tools and APIs?

A: They should test the full execution path, including prompts, retrieval, tools, API calls, and downstream side effects. A model that looks safe in isolation can still create risk once it can act across systems. The goal is to find where the workflow escapes its intended boundaries before attackers do.

Q: Why do traditional security controls struggle with agentic AI workflows?

A: Traditional controls assume predictable inputs, fixed behaviour, and static attack surfaces. Agentic systems break those assumptions because they can chain actions, change context, and produce different outcomes from similar inputs. Security teams need controls that follow runtime behaviour, not just configuration at deployment time.

Q: What do security teams get wrong about AI red teaming?

A: They often treat red teaming as a one-time model test instead of an ongoing system control. That misses the risk created by orchestration, tool integration, and changing production behaviour. Red teaming has to be continuous if the AI estate is continuously changing.

Q: How can organisations govern AI systems without slowing delivery?

A: By separating model evaluation from operational access governance. Teams should permit experimentation in controlled environments, then gate production tool access, data reach, and action authority behind explicit review and monitoring. That lets development continue while reducing the chance that an AI workflow can cause unintended real-world effects.

Technical breakdown

Full-stack AI red teaming and the seam problem

Modern AI red teaming has moved beyond isolated prompts and model responses. The article frames AI systems as composed environments where models, retrieval layers, agents, APIs, and infrastructure interact. That composition creates seam risk, meaning the failure often appears between components rather than inside one component. Red teams therefore test how inputs propagate across the stack, how context is preserved or corrupted, and how tool calls behave under adversarial pressure. This is a system-integrity problem, not a single-control problem.

Practical implication: Test the full execution path, not only model outputs, so you can see where privilege, data access, and orchestration controls break down.

Agentic workflows and tool integrations as attack surfaces

Agentic systems change the risk model because they do more than answer questions. They can select tools, chain steps, and operate across multiple decision points, which means a small prompt manipulation can become a real-world action. That raises the importance of tool authorization, scoped data access, and execution boundaries. The article’s core message is that traditional assumptions about fixed input and predictable output do not hold once AI can initiate actions across environments. The attacker target becomes behaviour, not just content.

Practical implication: Treat every tool connection as an access path that needs explicit scoping, monitoring, and revocation logic.

Why deterministic security assumptions fail for AI

The article states that conventional security frameworks were built for deterministic systems with predictable inputs and static attack surfaces. AI systems violate that model because their behaviour is probabilistic, context-sensitive, and continuously evolving. That means static policy checks and one-time validation cannot provide durable assurance. Red teaming becomes an ongoing control, not a pre-launch exercise. The key architectural point is that security has to observe behaviour over time, especially where the same input can produce different outcomes.

Practical implication: Build continuous testing and runtime monitoring into AI operations, because point-in-time approval does not cover changing behaviour.

Threat narrative

Attacker objective: The attacker aims to turn a normal AI workflow into an action path that produces unintended business, data, or infrastructure outcomes.

1. Entry occurs when an adversary manipulates prompts, retrieval inputs, or connected data sources to influence an AI workflow.
2. Escalation follows when the system chains tool use or external actions beyond the original intent of the request.
3. Impact occurs when the manipulated workflow produces unauthorized decisions, data exposure, or operational side effects across connected systems.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

The seam is now the primary attack surface in AI security. The article is right to move red teaming from model-only testing to full-stack simulation because risk increasingly appears where retrieval, tools, agents, and deployment layers interact. That is where system integrity fails first, especially when one component trusts another too easily. For practitioners, the implication is clear: security review has to follow the interaction path, not the component list.

Deterministic security assumptions do not survive agentic behaviour. The article shows that once AI systems can chain steps and call tools, static input-output thinking stops being sufficient. A control that assumes predictable outputs cannot explain a workflow that adapts at runtime, draws on external data, and executes across environments. The implication is that security architecture must account for behaviour that changes as the system runs.

Model testing alone is a narrow control for an operational AI estate. The article describes a shift toward continuous adversarial simulation because production risk now lives in orchestration, integration, and runtime access. That expands the governance problem from accuracy to accountability, privilege, and containment. Practitioners should treat AI red teaming as an operational discipline, not a pre-deployment checklist.

AI red teaming is becoming an identity control problem as much as a model-control problem. When an AI system can act through external tools, the relevant question is no longer only what the model says, but what the system is authorised to do. That brings NHI governance, access boundaries, and runtime oversight into the same conversation as adversarial testing. Teams need to govern the credentials and permissions behind the workflow, not just the model layer.

Autonomous action changes the failure mode from bad output to compounded execution. This article shows how agentic systems can amplify minor manipulations into multi-step consequences across connected systems. The implication is that organisations must evaluate escalation paths, not only initial prompt injection risk, because the harm often comes from what the workflow does after the first decision.

From our research:

• 70% of organisations grant AI systems more access than they would give a human employee performing the exact same job, according to the 2026 Infrastructure Identity Survey.
• Only 44% of organisations have implemented any policies to manage their AI agents, even though 92% agree that governing AI agents is critical to enterprise security.
• That gap points to the next step for teams, which is to align agent governance with the 2026 OWASP Top 10 for agentic applications before runtime access becomes the default control problem.

What this signals

Agentic workflow governance: the issue is not only whether an AI system is accurate, but whether its runtime access is bounded well enough to stay inside policy. The article suggests that red teaming needs to move closer to identity and authorization review, because tool use is where business impact is created.

With 70% of organisations already granting AI systems more access than they would give a human employee performing the exact same job, per the 2026 Infrastructure Identity Survey, the governance gap is no longer theoretical. Security teams should expect pressure to prove where agent permissions are defined, reviewed, and revoked.

The practical signal for programmes is that AI security testing, IAM, and NHI governance are converging around the same runtime questions. Teams that keep those functions separate will miss the point where adversarial behaviour becomes operational behaviour.

For practitioners

• Map the full AI execution path Inventory prompts, retrieval layers, tool calls, APIs, and deployment dependencies so red teaming covers the complete chain rather than a single model endpoint.
• Scope tool access as identity, not convenience Review every agent-connected tool for explicit authorization boundaries, least-privilege access, and revocation authority before the system can reach production data or controls.
• Run continuous adversarial simulations Schedule red-team tests that exercise prompt manipulation, context poisoning, tool misuse, and orchestration drift in the same environment where the system runs.
• Monitor runtime behaviour and side effects Track actual tool usage, data access, and action outcomes in production so changes in system behaviour are visible before they become incidents.

Key takeaways

• AI red teaming has moved from model probing to full-stack adversarial simulation because the real risk now appears where systems interact.
• Agentic workflows expand the attack surface by turning small manipulations into chained actions that can cross tools, APIs, and environments.
• Security teams need continuous testing, scoped tool access, and runtime monitoring if they want AI systems to remain governable in production.

Key terms

• Agentic Workflow: An agentic workflow is an AI-driven process that can select tools, chain actions, and progress through multiple steps with limited human intervention. In security terms, the governance question is not just what the model says, but what it is allowed to do at runtime and how those permissions are constrained.
• Full-stack Red Teaming: Full-stack red teaming tests the complete AI system, including models, retrieval, integrations, infrastructure, and deployment paths. The purpose is to find failure points where component interactions create risk that would not appear in a model-only assessment.
• Runtime Access Boundary: A runtime access boundary is the set of permissions, tools, and data paths an AI system can use while it is operating. For AI and NHI governance, it defines the difference between controlled execution and a workflow that can reach beyond its intended scope.
• System Integrity: System integrity is the degree to which interconnected components behave as designed under normal and adversarial conditions. In AI security, it means the workflow remains trustworthy across orchestration, context changes, and external tool use, not merely that a model responds correctly in testing.

What's in the full article

TROJ.AI's full blog covers the operational detail this post intentionally leaves for the source:

• The webinar discussion points from Lee Weiner, John Vaina, and Gavin Klondike on how red teaming evolved over the past year
• The full breakdown of full-stack testing across models, retrieval, tool integrations, infrastructure, and deployment environments
• The practitioner-oriented examples of adversarial simulation that sit behind the article's broader claims
• The source team's framing of continuous AI security as a development lifecycle discipline rather than a one-time review

👉 TROJ.AI's full post covers the webinar themes, system-level risk framing, and production security implications.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.