What breaks when agent behaviour is not tested adversarially?

Why This Matters for Security Teams

Agent behaviour that is never tested adversarially tends to fail in the same places attackers look first: prompt injection, tool abuse, unsafe retrieval, and over-broad action authority. That matters because these are not model-only defects. They become operational defects when an agent can read, decide, and act on behalf of a user or workflow. The relevant threat patterns are described in MITRE ATLAS adversarial AI threat matrix and in current guidance from the NIST AI Risk Management Framework.

Security teams often underestimate the distance between a “bad answer” and a “bad action.” An agent that cites the wrong source is annoying; an agent that opens a ticket, sends a message, changes a record, or executes a tool call can create real business impact. Adversarial testing is what exposes whether guardrails actually constrain those action paths, or merely shape the wording of the model output. In practice, many security teams encounter abuse only after a workflow has already been treated as trusted automation rather than through intentional red-team testing.

How It Works in Practice

Adversarial testing should cover the full agent loop: input, retrieval, reasoning, tool selection, execution, and post-action validation. The goal is not to “break the model” in the abstract. It is to prove how the system behaves when the prompt is manipulated, retrieved content is malicious, context is poisoned, or a tool returns unexpected instructions. That is why frameworks such as the OWASP Agentic AI Top 10 and CSA MAESTRO agentic AI threat modeling framework emphasise agent-specific attack surfaces rather than generic app testing.

• Test prompt injection against both system prompts and user-visible prompts.
• Probe tool permissions, including whether the agent can exceed intended scope.
• Inject hostile or misleading retrieval content and verify output validation.
• Check whether the agent can be coerced into leaking secrets, tokens, or hidden instructions.
• Simulate chained abuse, where one weak step leads to a harmful downstream action.

For practitioners, the most useful test cases resemble attacker behaviour observed in the wild. Public reporting on the first AI-orchestrated cyber espionage campaign from Anthropic shows how agentic systems can be steered into operational misuse when controls are weak. These controls tend to break down when agents are connected to legacy workflows with broad API permissions and no separate approval gate for high-impact actions.

Common Variations and Edge Cases

Tighter adversarial testing often increases release friction, requiring organisations to balance coverage against velocity and the cost of maintaining test harnesses. There is no universal standard for how much adversarial coverage is enough, so best practice is evolving toward risk-based testing depth rather than one fixed checklist. High-impact agents deserve deeper testing than low-risk assistants, especially where external tools, regulated data, or privileged workflows are involved.

One common edge case is an agent that appears safe in isolation but fails when chained with retrieval, memory, or downstream automation. Another is the “helpful fallback” problem, where the agent bypasses a failed control by improvising an alternative action path. Security teams should also distinguish between prompt-level resilience and operational resilience. A response that refuses unsafe instructions is useful, but it is not sufficient if the same request can still trigger a harmful tool call.

For identity-heavy deployments, adversarial testing should also consider who the agent is acting as, what authority is delegated, and whether those permissions are time-bound and auditable. That becomes especially important when an agent can initiate access, move data, or impersonate a trusted workflow. In those cases, current guidance suggests combining AI testing with identity controls from NIST SP 800-63 Digital Identity Guidelines and incident review signals from CISA cyber threat advisories.

Related resources from NHI Mgmt Group

• What breaks when agent behaviour drifts beyond approved scope?
• What breaks when agent behaviour is monitored only at the platform layer?
• What breaks when AI agent behaviour is only monitored at the prompt layer?
• What breaks when sandbox validation does not match actual execution in agent systems?