Model-only testing misses the attack paths that emerge once an agent can call tools, traverse MCP connections, and carry context across multiple turns. In practice, the failure is a false sense of safety: prompt checks can pass while tool abuse, connector poisoning, or chained actions still create real compromise risk.
Why model-only red teaming gives a misleading result
Testing only the model layer answers a narrower question than most teams think it does. It shows whether the model can resist bad prompts in isolation, but it does not show whether the system stays safe once the model is embedded in a tool-using workflow, a multi-step orchestration path, or an agent that can persist context and act across turns.
That is the core break: the security boundary moves from “can the model be tricked?” to “can a chained system be driven into doing something harmful?” When the red team ignores tool calls, connectors, and stateful memory, it can miss the real attack surface entirely, especially in agentic designs where action, not text, is the risk-bearing output.
Where the attack surface expands beyond the model
A model-only test usually exercises prompt injection, jailbreak resistance, and content filtering. Those checks matter, but they do not cover the mechanisms that create compromise once the model can act. Tool invocation, connector trust, delegated permissions, retrieval sources, and cross-turn memory can all turn a “safe” response into an unsafe sequence of actions.
This is why MCP connections and similar integrations are so important to include in red teaming. Once the model can traverse external systems, an attacker may not need to win the prompt battle at all. They may instead poison a connected source, steer the agent through a malicious instruction path, or abuse a legitimate tool to trigger unauthorized reads, writes, or approvals.
The practical failure mode is often chained behavior. A single prompt may look harmless, but the combination of tool selection, retrieved context, and follow-up actions can produce escalation, data exposure, or workflow abuse that no isolated model test would reveal. For agentic systems, the question is not only “what did the model say?” but “what was the system able to do as a result?”
What a complete red team must exercise
Effective ai red teaming should include the end-to-end path the user actually experiences. That means testing the model, the tools it can call, the connectors it trusts, the context it retains, and the policies that govern execution. A system can pass text-only checks and still fail once the agent can reach external services, reuse prior context, or chain actions across multiple turns.
- Test tool misuse, not just prompt misuse, because action privileges often define the real blast radius.
- Test connector poisoning and retrieved-context manipulation, because untrusted inputs can shape later decisions.
- Test multi-turn persistence, because abuse often emerges only after the agent has accumulated state.
- Test approval and delegation boundaries, because a model that can request action is not the same as one that can safely perform it.
Red teaming should also treat the orchestration layer as part of the target. If the agent can reach a browser, database, ticketing system, repository, or internal API, then the assessment has moved into identity abuse in AI agents, not just model robustness. For teams building a more structured program, the broader control and maturity questions are covered in the Agentic AI Identity Maturity Model, which helps separate model safety from system authority.
Risk and Threat Considerations
Model-only red teaming creates false confidence because the most damaging failures often happen after the model has already produced a plausible output. Attackers can target the tools, connectors, memory, or orchestration around the model, then use legitimate execution paths to cause unauthorized access, data leakage, or destructive actions.
Failure mechanism: The test scope excludes the parts of the system that convert text into action, so it never exercises prompt injection through tools, connector poisoning, delegated privilege abuse, or multi-turn escalation paths.
Impact: Teams may ship an agent that appears robust in evaluation but remains exploitable in production, where the real risk is not the response text itself but the downstream side effects of tool use and chained execution.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST SP 800-53 Rev 5 sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI02 — Tool Misuse | Directly addresses unsafe agent actions through tools and connectors. |
| ASI03 — Identity & Privilege Abuse | Covers delegated authority and privilege abuse once agents can act. | |
| Recommendation — Test and restrict agent tool use to prevent harmful or unauthorized actions. Bound agent privileges and validate each delegated action before execution. | ||
| MITRE ATLAS | Adversarial AI threat matrix | Supports red teaming of AI attack paths, prompt injection, and context poisoning. |
| Recommendation — Map agent attack scenarios to adversarial AI techniques and test the full chain. | ||
| NIST SP 800-53 Rev 5 | AC-6 — Least Privilege | Limits blast radius when agents can invoke tools or reach connected systems. |
| AU-6 — Audit Record Review, Analysis, and Reporting | Needed to detect suspicious tool calls and chained agent activity. | |
| Recommendation — Apply least privilege to every agent-facing account, token, and connector. Review agent execution logs for abnormal tool use and escalation patterns. | ||
Practitioner Guidance
What to verify: Confirm that red team scenarios cover every place the model can influence state, including tool calls, retrieval sources, memory, connectors, approval steps, and any external action channel. If a path can change data, permissions, or workflow state, it belongs in scope.
Decision rule: If the agent can do anything beyond answering in plain text, treat model-only testing as incomplete and use system-level scenarios that simulate real attacker progression, not isolated prompt failures.
What good looks like: A strong program can show not only that prompts are resisted, but also that unsafe actions are blocked, suspicious tool use is contained, and cross-turn abuse does not accumulate into a successful compromise.
Practitioner takeaway: Red teaming that stops at the model layer measures persuasion resistance, not operational safety; the security verdict only becomes meaningful when the agent’s tools, memory, and authority are tested together.