AI agent red teaming exposes the trust debt in autonomous systems

At a glance

What this is: This is Fiddler's analysis of AI agent red teaming, showing that agent testing must focus on real actions, tool use, and multi-step abuse paths rather than text-only safety checks.

Why it matters: It matters because IAM, PAM, and AI governance teams now have to control what agents can access, do, and chain together across systems, not just whether an LLM output looks safe.

By the numbers:

• Roleplay-based prompt injection attacks achieved an 89.6% attack success rate, outperforming logic traps at 81.4% and encoding tricks at 76.2%.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, sharing sensitive data, or revealing credentials.

👉 Read Fiddler's analysis of AI agent red teaming techniques and attack surfaces

Context

AI agent red teaming is a control problem, not a prompt-quality problem. Once an agent can call tools, query data, and act across systems, the risk shifts from bad text to unsafe execution, and traditional guardrails stop being enough. That creates a direct governance issue for agentic AI programmes, where access boundaries and approval logic must be treated as security controls.

The primary weakness is trust debt, the gap between the autonomy granted to an agent and the organisation's ability to monitor or constrain it. In practice, that gap grows whenever teams add tools, connect more data sources, or allow agents to chain steps without strict policy enforcement. For identity programmes, this is where NHI governance and AI governance converge.

Fiddler's starting point is typical for organisations adopting agents quickly: capability grows first, oversight follows later, and adversarial testing becomes the only reliable way to see where the controls actually fail.

Key questions

Q: How should security teams red team AI agents that use tools and memory?

A: Security teams should test AI agents through the same interface and runtime path production uses, then validate the tools, memory stores, and downstream sinks those agents can reach. A good program ties each finding to an observed side effect, such as a webhook call, data write, or workflow trigger, rather than treating prompt success or failure as the result.

Q: What breaks when AI agents are not governed at runtime?

A: Without runtime governance, an agent can shift behaviour after provisioning and still execute actions that were never reviewed in context. That is where tool chaining, MCP connections, and rapid decision-making become dangerous. Static approval cannot stop a live change in intent, so teams lose control at the point of action.

Q: How do security teams know if AI red teaming is working?

A: AI red teaming is working when testing finds real prompt injection paths, over-scoped integrations, and policy gaps before attackers do, and when fixes are re-tested successfully after model or workflow changes. The strongest signal is repeatable reduction in exposed authority, not a lower number of red-team findings on its own.

Q: When should organisations treat an AI agent as a privileged system?

A: Organisations should treat an AI agent as privileged whenever it can reach production data, administrative tools, or sensitive workflows without direct human approval for each step. At that point, the agent is no longer a passive automation helper. It becomes a governed identity whose permissions, logs, and exceptions need the same scrutiny as other high-risk access.

Technical breakdown

Why prompt injection remains the dominant agent attack surface

Prompt injection succeeds because agents do not merely parse language, they use language as a control input for later actions. Direct injection places malicious instructions in the current prompt, while indirect injection hides them in retrieved content such as documents or emails. Roleplay-based attacks work especially well because they manipulate context and persona, causing the agent to treat attacker instructions as legitimate. Keyword filters fail here because the harmful intent is often embedded indirectly, and the execution happens in downstream tool calls rather than in the response itself.

Practical implication: Test for semantic manipulation and role confusion, not only blocked words, and enforce policy checks before any tool invocation.

How tool invocation turns a model error into an operational incident

An agent becomes materially different from a chat model when it can invoke email, database, ticketing, or payment tools. The risk is not just that the model says the wrong thing, but that it can execute a sequence of authorised-looking steps that produces an unauthorised outcome. This is why chained actions matter: each step may look harmless in isolation, yet the full sequence can extract data, transform it, and exfiltrate it. That is a policy failure in the orchestration layer, not a language-model failure alone.

Practical implication: Constrain each tool with explicit scope, record decision lineage, and block multi-step chains that exceed a task's approved boundary.

Why trust boundaries and RAG content need continuous review

Retrieval-augmented generation creates a persistent attack surface because poisoned content can influence future sessions long after initial ingestion. If an attacker plants instructions in a support ticket, knowledge base, or indexed document, the agent may retrieve that content repeatedly and act on it as if it were trusted context. That turns one successful injection into a durable control-plane problem. Trust boundaries therefore need to be mapped across data sources, retrieval logic, and tool permissions, not just at the model interface.

Practical implication: Treat retrieved content as untrusted until validated, and monitor knowledge sources for hidden instructions or abnormal influence patterns.

Threat narrative

Attacker objective: The attacker wants the agent to carry out unsafe actions on their behalf, including data exposure, unwanted communications, or policy-bypassing tool use.

1. Entry occurs when attackers place malicious instructions into user prompts, retrieved documents, emails, or other context sources that the agent consumes during runtime.
2. Escalation follows when the agent treats those instructions as legitimate, gains access to tools or data it should not use, and chains actions across connected systems.
3. Impact occurs when the agent exposes data, sends messages, or performs another harmful action that passes ordinary safety checks but violates business policy.

NHI Mgmt Group analysis

AI agent red teaming is now an identity governance problem as much as a model testing problem. Once an agent can access data, call tools, and chain decisions, the central question becomes who or what is authorised to act. That is an NHI governance issue because the agent operates as a software entity with delegated access that must be bounded, reviewed, and revoked like any other machine identity. Practitioners should stop treating agent testing as a niche AI exercise and fold it into IAM and PAM controls.

Trust debt is the most useful concept for understanding agent risk. It describes the growing gap between the autonomy an organisation grants and the actual ability to supervise that autonomy. The article's core point is that this gap widens as teams add tools and data sources faster than they add policy enforcement. That makes trust debt a programme-level risk indicator, not just a testing observation, and it should influence access design from the start.

Prompt injection is no longer just an input-validation issue, it is a control-plane exploit path. Roleplay and indirect injection work because they steer the agent's internal decision process rather than simply corrupting text output. That means the real failure is not a missing filter, but a weak boundary between interpreted context and executable action. Security teams should frame this as policy enforcement at the point of action, which is where NHI and agentic AI governance overlap most clearly.

Continuous red teaming is the only defensible response to changing agent behaviour. Agent capabilities shift when models are updated, tools are added, or retrieval sources change, so yesterday's safe path can become tomorrow's exploit path. Static approval models do not keep pace with that drift. The practical conclusion is that agent governance needs recurring adversarial validation, just as identity programmes need recurring access review and entitlement control.

Runtime guardrails should be evaluated as identity controls, not just safety features. If a guardrail cannot stop an unauthorised tool call, it has failed to enforce privilege boundaries. That places it closer to access control and segregation-of-duties than to content moderation. The field should treat agent guardrails as part of the broader delegated-authorisation stack, where least privilege and policy enforcement have to operate continuously.

What this signals

Trust debt will become the practical measure of agent governance maturity. As organisations connect agents to more tools, the deciding question will be whether access can be constrained and observed at the same pace as autonomy expands. The teams that win here will treat delegated action as a governed entitlement problem, not a novelty layer on top of application security.

Red teaming should now be built into the same operating rhythm as access review and change control. If a model update, new tool, or new retrieval source can change agent behaviour, then the validation cycle has to track those events continuously. That is especially true where agent identity, service accounts, and API credentials form part of the execution path.

The useful shift is from testing for harmful text to testing for harmful execution. That means policy, telemetry, and entitlement design must sit together in the programme, with control evidence mapped back to frameworks such as the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10.

For practitioners

• Map agent authority before testing begins Document every action an agent can take, every tool it can call, and every data source it can reach. Separate read-only access from action-taking permissions, and treat each autonomous path as an entitlement that must be justified and reviewed.
• Prioritise roleplay and indirect injection scenarios Build red team cases that use persona shifts, hidden instructions in retrieved content, and misleading context instead of simple keyword abuse. These scenarios better reflect how real attackers bypass guardrails and they expose policy gaps that basic tests miss.
• Enforce runtime policy checks at the point of action Block or approve tool invocations before execution, not after the fact. The policy engine should evaluate destination, data sensitivity, task scope, and sequence context so chained actions cannot cross authorised boundaries unnoticed.
• Monitor decision lineage and tool-use drift continuously Track why the agent selected an action, which tools were used, and whether those patterns change after model updates or new integrations. Feed anomalies back into the red team programme so production findings become new adversarial tests.

Key takeaways

• AI agent risk is about execution, not output, because a safe-seeming response can still trigger an unsafe tool chain.
• Trust debt grows when autonomy expands faster than oversight, which is why red teaming has to be continuous rather than episodic.
• Runtime guardrails and decision-lineage monitoring are the controls that turn red team findings into production protection.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Trust debt: Accumulated security risk created when access assumptions are not revalidated quickly enough for the pace of modern automation. In identity programmes, trust debt appears when roles, secrets, or agent permissions persist longer than the environment that justified them.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Context Poisoning: Context poisoning is the manipulation of information that an AI agent reads before acting. The malicious content does not need to be code. If it changes the agent’s instructions, tool choices, or assumptions, it can alter behaviour and expand the impact of a compromised delivery path.

What's in the full article

Fiddler's full blog post covers the operational detail this post intentionally leaves for the source:

• Step-by-step red team structure for macro-level and micro-level agent testing
• Attack scenario examples for prompt injection, tool abuse, and context poisoning
• Guidance on converting findings into runtime policy enforcement and continuous monitoring
• Practical advice on scheduling recurring exercises as agent capabilities change

👉 Fiddler's full post covers attack techniques, testing structure, and runtime guardrail design in more operational detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management for practitioners who need to control delegated access. It helps security teams connect identity boundaries to the wider governance model their programmes depend on.