AI red teaming now spans the full AI stack and agentic risk

At a glance

What this is: This blog argues that AI red teaming now has to cover the full stack, because risk increasingly appears in how models, tools, retrieval, and deployment layers interact under adversarial pressure.

Why it matters: For IAM and security teams, the key issue is that agentic AI changes the identity problem from static access control to runtime behaviour, tool invocation, and cross-system trust.

Context

AI red teaming now has to examine more than a model’s output quality. When AI systems can call tools, traverse retrieval layers, and trigger actions in connected environments, the security question becomes how the whole system behaves under pressure.

TROJ.AI’s article treats AI security as an operational discipline rather than a lab exercise. The governing problem is no longer just prompt injection or model misuse in isolation, but the interaction between probabilistic behaviour, connected systems, and autonomous execution paths.

That makes the article relevant to agentic AI governance, where identity, authorization, and runtime controls have to be evaluated together rather than as separate design assumptions.

Key questions

Q: How should security teams use red-team style challenges to improve AI prompt injection defenses?

A: Security teams should use controlled adversarial testing to expose how an AI system reacts to unsafe prompts, data leakage attempts, and instruction overrides. The goal is not just to block attacks, but to learn where controls fail, how users behave, and which defenses need tuning. Repeated testing helps teams measure resilience, improve guardrails, and align AI security with real attacker techniques.

Q: Why do AI systems create identity risk as well as model risk?

A: Because AI systems rarely act alone. They depend on service accounts, API tokens, cloud permissions, and data access paths, which means a model can behave safely while its identity layer is over-privileged. Treating AI risk as only a model problem misses the access surface where misuse and lateral movement usually begin.

Q: What are the signs that AI assurance is too narrow?

A: The clearest sign is when teams only test model accuracy or prompt behaviour while ignoring data integrity, provenance, access control and monitoring. That usually means the programme is validating outputs but not the operational conditions that make those outputs trustworthy. Assurance needs to cover the full path from data input to deployment.

Q: How can organisations limit the impact of autonomous AI actions?

A: By constraining which tools a system may invoke, limiting the systems it can reach, and requiring review for high-impact steps. The goal is to keep a single AI action from propagating into multiple downstream changes across environments.

Technical breakdown

Full-stack red teaming across models, retrieval, and workflows

Full-stack AI red teaming tests how components behave together, not just whether a model can be manipulated in isolation. That includes prompts, retrieval systems, data pipelines, agentic workflows, and deployment environments. The reason this matters is simple: modern AI failures often emerge at integration points, where one layer accepts untrusted input and another layer turns that input into action. Traditional test cases miss those cross-layer dependencies because they assume a fixed boundary between input validation, model inference, and execution. In practice, security teams need to treat the stack as a single attack surface, not a set of disconnected components.

Practical implication: Map red-team scenarios to the full AI stack, including retrieval, tool access, and runtime execution paths.

Why agentic AI changes the identity and access model

Agentic systems are not just smarter automation. They can take independent actions, interact with tools and APIs, and chain decisions across steps. That changes the identity problem because the system is no longer only producing content, it is acting. Once a model can invoke external functions or continue a workflow without human approval at each step, access control becomes a runtime governance issue rather than a static provisioning issue. The article’s central point is that prompt-level controls do not contain the full risk surface when autonomy is introduced. The real concern is how authority is delegated, constrained, and observed during execution.

Practical implication: Review where AI systems can act on delegated authority without a fresh authorization decision.

Deterministic security assumptions do not survive autonomous tool use

Many conventional controls assume predictable inputs, stable behaviour, and bounded outcomes. Agentic AI breaks those assumptions because the same input can lead to different actions depending on context, tool availability, and prior steps in the workflow. That makes static policy checks less reliable, especially when small prompt changes can cascade into external actions with material impact. The technical issue is not merely model hallucination. It is the combination of context sensitivity, runtime decision-making, and chained tool access. Security architecture has to account for behaviour that evolves during the session, not only at provisioning time.

Practical implication: Build controls that inspect behaviour during execution, not just content before submission.

Threat narrative

Attacker objective: The attacker aims to convert a seemingly minor interaction into unauthorized system action by abusing autonomy, tool access, and cross-layer propagation.

1. Entry occurs when adversarial prompts, malformed retrieval content, or manipulated workflow inputs reach the AI stack through normal application channels.
2. Credential or authority abuse follows when an agentic system uses delegated tool access or API connectivity to carry out actions beyond the intended user request.
3. Escalation happens as a small input manipulation propagates across chained decisions, allowing one interaction to influence multiple downstream systems.
4. Impact is produced when the AI system turns that chained behaviour into real-world action, including changes in external environments or mission-critical workflows.

Breaches seen in the wild

• Langflow Flodrix botnet 2025: Attackers used Langflow's unauthenticated code endpoint to dump AI servers' environment variables and enlist them in the Flodrix botnet.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Full-stack AI red teaming is now the only defensible unit of analysis: model-only testing misses the seams where retrieval, workflow logic, tool access, and deployment behavior combine into an exploitable path. The article correctly reflects that risk emerges in composition, not just in model behaviour. Security teams should stop treating the model as the boundary and start treating the entire AI application as the attack surface.

Agentic AI turns authorization into a runtime governance problem: once a system can select tools and continue actions across steps, access is no longer just a provisioning decision. The governance question shifts to who can trigger actions, under what context, and with what observable constraints. Practitioners should interpret agentic workflows as delegated execution, not simply advanced inference.

Deterministic security assumptions are the broken premise: static inputs, stable outputs, and predictable execution were designed for non-agentic systems. That assumption fails when AI systems can adapt mid-session, chain decisions, and amplify a small prompt manipulation into an external action. The implication is that review and approval models built for fixed workflows will not describe the actual risk surface.

Continuous adversarial simulation is becoming a baseline control, not a specialist exercise: the article shows that AI systems are now treated as operational infrastructure, which means security failures have production impact rather than lab impact. That places red teaming closer to ongoing control validation than one-time assessment. Practitioners should expect AI security testing to move into standard operational governance.

Identity blast radius: the material issue is not just whether an AI agent is authorised, but how far that authorisation propagates once the system begins chaining tools, decisions, and environments. That creates a new control question for IAM and PAM teams: how much downstream effect a single delegated action can generate. The programme response has to be built around limiting blast radius, not assuming the model will self-contain.

What this signals

AI red teaming now has to follow the workflow, not just the model: the next governance step is to test where a seemingly safe prompt becomes a tool call, a retrieval lookup, or an external write operation. That is the point where AI security starts overlapping with identity governance.

Traditional access controls still matter, but they are no longer sufficient on their own when AI systems can execute chained actions. The practical shift is toward runtime observability, delegated authority limits, and tighter boundaries around what an agent may touch during a session.

For practitioners

• Expand red-team scope to the full AI stack Test prompts, retrieval systems, workflow logic, agent tools, and deployment layers in one scenario set so the evaluation reflects real system composition.
• Inventory every delegated tool path Map which AI systems can call APIs, write to external systems, or trigger multi-step workflows without a human approval gate in between.
• Separate content safety from execution safety Treat prompt filtering and model output review as insufficient unless the runtime path that turns output into action is also constrained.
• Add behavioural monitoring in production Watch for tool calls, chain length, unusual sequencing, and repeated retries that indicate the system is acting outside its expected operating pattern.
• Define blast-radius limits for agentic actions Set explicit boundaries on which systems an AI workflow may touch, then review whether those boundaries still hold once the system starts chaining decisions.

Key takeaways

• AI red teaming has moved into system composition, because the most important failures now emerge where models, retrieval, tools, and infrastructure interact.
• Agentic AI turns authorization into a runtime issue, since delegated actions can propagate across multiple steps and environments.
• Security teams need to test behaviour under adversarial conditions and constrain blast radius, not rely on model-level checks alone.

Key terms

• Agentic AI Red Teaming: Agentic AI red teaming is the practice of testing autonomous AI systems for unsafe behavior, misuse, and control failures. It uses adversarial scenarios to probe how an agent plans, acts, calls tools, handles prompts, and responds to manipulation, with attention to identity, authorization, data exposure, and unintended actions.
• Full-Stack AI Security: A security approach that evaluates the entire AI application, including prompts, models, retrieval, workflows, APIs, and deployment infrastructure. The goal is to find system-level failures that only appear when components are composed and the AI system is allowed to act in production.
• Delegated Execution: Delegated execution is when software is allowed to perform actions on behalf of a user, process, or business function. In NHI governance, the risk is that the delegated actor may chain actions beyond the original intent, so controls must focus on scope, approval, and revocation.
• Identity Blast Radius: The amount of damage a compromised identity can cause across systems, data, and infrastructure. In NHI environments, it is shaped by permissions, network reach, and administrative capability rather than by the credential alone. Reducing blast radius is a containment strategy that limits lateral movement and data exposure.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.