What breaks when agentic systems are tested only at the model level?

Model-level testing misses the dangerous part of the system, which is how the agent behaves once it can use tools, remember context, and chain actions across sessions. A prompt that looks harmless in isolation can become a harmful instruction when combined with a tool response or external data source.

Why This Matters for Security Teams

Testing only the model creates a false sense of safety because the agent’s real risk emerges after it is connected to tools, data, memory, and orchestration logic. That is where prompt injection becomes operational, where benign-looking outputs can trigger actions, and where tool calls can amplify a single bad instruction into a multi-step failure. The security question is not just what the model says, but what the system is allowed to do next. The OWASP Agentic AI Top 10 captures this shift clearly: agentic risk is system risk, not just model risk.

Practitioners often miss that evaluations focused on static prompts do not exercise session state, retrieval paths, privilege boundaries, or tool authorization. A model can look robust in a lab and still fail when a connector returns hostile content, a planner retries a task, or memory persists unsafe context across sessions. Current guidance suggests evaluating the full action chain, including input handling, reasoning steps, tool invocation, and post-action validation. In practice, many security teams encounter agentic failure only after an external data source or tool response has already been trusted as if it were ground truth.

How It Works in Practice

Effective testing treats the agent as an end-to-end workflow with multiple trust boundaries. That means creating scenarios that combine prompt manipulation, retrieval poisoning, tool abuse, and authorization edge cases rather than scoring isolated model outputs. The NIST AI Risk Management Framework is useful here because it pushes teams toward governance, measurement, and ongoing monitoring instead of one-time validation.

A practical test plan usually includes:

• Prompt injection cases that arrive through chat, files, web pages, tickets, or retrieved context.
• Tool-chain abuse cases where the agent is nudged into calling email, payment, deployment, or ticketing actions.
• Memory and session tests to see whether unsafe instructions persist or compound over time.
• Permission boundary tests that verify the agent cannot exceed the user’s or workflow’s authority.
• Output validation tests that check whether the system confirms actions before execution.

Security teams should also align tests to known attack patterns. MITRE ATLAS adversarial AI threat matrix helps map evasion, poisoning, and manipulation behaviours, while the CSA MAESTRO agentic AI threat modeling framework is helpful for modelling how planners, tools, and memory interact under adversarial conditions. The key is to validate what the system does when it is pressured, not just whether the underlying model gives a safe answer in isolation. These controls tend to break down when tool permissions are broad and the agent can act across multiple services without a hard approval step because one compromised context can cascade into several trusted actions.

Common Variations and Edge Cases

Tighter agent testing often increases engineering overhead, requiring organisations to balance coverage against development speed and operational complexity. That tradeoff is real, especially in environments where agents support customer operations, software delivery, or internal knowledge work.

There is no universal standard for this yet, but current guidance suggests the riskiest edge cases are not the obvious jailbreaks. They are the combined scenarios: a retrieved document containing hostile instructions, a long-lived memory entry that survives task completion, or a tool response that changes the agent’s plan mid-execution. The Anthropic first AI-orchestrated cyber espionage campaign report is a useful reminder that agentic abuse becomes more serious when systems can chain actions across contexts.

Another common gap appears when teams test only the “happy path” in controlled demos. That misses degraded network states, partial tool failures, stale permissions, and noisy retrieval results, all of which can change agent behaviour. For higher-risk use cases, practitioners should separate model evaluation, tool safety testing, and workflow approval testing rather than treating them as one control. Best practice is evolving, but the operational rule is stable: if the agent can take an action, then the test must include the conditions under which that action becomes unsafe.

Related resources from NHI Mgmt Group

• What breaks when model-level guardrails are treated as security controls for AI systems?
• What breaks when agentic systems outgrow manual documentation?
• What breaks when identity is used as the only control for agentic systems?
• What breaks when retrieval happens before authorization in agentic AI systems?