How should security teams red team agentic AI systems differently from standalone chatbots?

Why Red Teaming Agentic AI Is Different

Standalone chatbots are usually judged on answer quality, policy compliance, and jailbreak resistance. Agentic systems need a broader test because the real risk is not just what they say, but what they can do after they answer. A red team should trace tool use, memory, approval flows, and inter-agent handoffs as part of one attack path. That is why guidance around agentic systems increasingly treats the workflow as the target, not the prompt.

NHIMG’s analysis of agentic exposure and compromise patterns shows why this matters: the attack surface expands once an AI system can act on credentials, data, or downstream tools rather than simply generate text. Public reporting on the AI Agents: The New Attack Surface report and the OWASP Agentic AI Top 10 both point to the same issue: agents can chain actions in ways that are hard to predict from a single prompt exchange. In practice, teams often discover the weakness only after an agent has already queried a connected system, reused a token, or passed unsafe instructions into another service.

What to Test in the Agent Workflow

Red teaming should start with the question, “What can the agent reach, change, or leak?” rather than “Can the model be tricked?” The evaluation should cover tool definitions, memory stores, policy checks, retrieval layers, and any orchestration layer that turns model output into action. Current guidance suggests testing both direct attacks and indirect attacks, because malicious content may arrive through documents, tickets, emails, or shared context instead of a chat box.

Security teams should simulate abuse cases such as prompt injection that redirects the goal, privilege escalation through overly broad tool scopes, and data exfiltration through a legitimate workflow. They should also test whether the system can be coerced into unsafe multi-step behaviour, such as creating follow-on tasks, calling external APIs, or sharing secrets across agents. The CoPhish OAuth Token Theft via Copilot Studio case illustrates how an agentic workflow can become the real exploitation path, while the Anthropic report on AI-orchestrated cyber espionage shows how autonomy changes attacker tradecraft.

• Test prompt injection against both the model and any retrieved content.
• Validate tool permissions at runtime, not just in design reviews.
• Check whether memory can persist malicious intent across sessions.
• Confirm that inter-agent messages are authenticated, scoped, and logged.
• Measure whether the agent can be steered into unsafe actions without obvious policy violations.

These controls tend to break down when the agent is integrated with many SaaS tools, because the number of possible action chains grows faster than the red team can enumerate them.

Where Standard Chatbot Testing Breaks Down

Tighter testing often increases operational cost, because each agentic path may require a separate adversary scenario, evidence trail, and rollback plan. That tradeoff is unavoidable when the system has execution authority, but it also means security teams need to prioritise the highest-impact chains first.

For chatbots, a malicious prompt usually aims to change an answer. For agents, the same prompt may only be the first step in a longer compromise. Best practice is evolving, but there is no universal standard for this yet: teams often combine scenario-based red teaming, policy-as-code checks, and live monitoring of tool calls to cover the gap. Frameworks such as NIST AI Risk Management Framework and CSA MAESTRO agentic AI threat modeling framework are useful here because they force teams to think about governance, controllability, and failure impact, not just prompt safety. A separate lesson from the LLMjacking: How Attackers Hijack AI Using Compromised NHIs research is that compromised identities and exposed secrets can turn a well-behaved agent into an attacker-controlled workflow. Teams usually feel this gap only after an agent has already acted on behalf of a user or service account, not during a simple prompt-only test.

Related resources from NHI Mgmt Group

• How should security teams red team non-deterministic AI systems?
• How should security teams evaluate AI red teaming vendors for agentic systems?
• How should security teams evaluate AI code review tools for large repositories and multi-team workflows?
• How should security teams implement access control for generative AI systems without relying only on authentication?