What breaks when GenAI red teams test only individual model behavior instead of the full system pipeline?

Why This Matters for Security Teams

When GenAI red teams test only the model, they miss the thing attackers actually exploit: the pipeline around it. Prompt filtering, retrieval, tool calls, shared memory, approval steps, and downstream automation can turn a “safe” response into a harmful action. A model that looks resilient in isolation may still leak secrets, inherit bad context, or hand dangerous instructions to another service.

This is why NHI Management Group treats agentic risk as a system property, not a model property. The failure pattern is visible in real-world supply chain and workflow incidents, including the Reviewdog GitHub Action supply chain attack and the CI/CD pipeline exploitation case study, where compromise moved through orchestration layers rather than a single component. Current guidance also points to full-stack evaluation in the NIST AI 600-1 GenAI Profile.

In practice, many security teams discover pipeline-level exposure only after a benign-looking model test has already been promoted into a real workflow.

How It Works in Practice

Effective red teaming should follow the path of an actual task end to end. That means testing the model, the orchestrator, the retrieval layer, the tool executor, the identity boundary, and any human approval step in one scenario. A prompt that is harmless in a notebook can become dangerous once the system can search, write, approve, call APIs, or route work to another agent.

The practical question is not “Can the model answer safely?” but “What does the full system do when the model is wrong, manipulated, or overconfident?” That includes hidden prompt injection, context poisoning, tool misuse, and credential misuse through shared service accounts. The Anthropic Frontier Red Team analysis is useful here because it shows why model-centric testing can miss behaviour that emerges only once tools and instructions interact. NHI Management Group also recommends reviewing the DeepSeek breach to see how exposed data and system coupling can magnify downstream impact.

• Test prompt injection at the retrieval boundary, not just inside the model response.
• Verify whether tool calls are gated by policy, not by model judgment alone.
• Check whether one compromised agent can pass malicious context to another agent.
• Confirm secrets, tokens, and API keys are never available to the model unless strictly required.
• Replay the same attack across multiple workflow stages to see where it survives.

Red teams should document both direct model failures and system failures, because a weak handoff, permissive connector, or shared memory store can convert a partial issue into full workflow compromise. These controls tend to break down when agentic systems reuse context across tasks, because a single poisoned input can persist long enough to influence later tool use and approvals.

Common Variations and Edge Cases

Tighter end-to-end testing often increases cost and coordination overhead, requiring organisations to balance coverage against release speed. That tradeoff matters because not every GenAI system has the same blast radius. A standalone chatbot, a retrieval assistant, and an autonomous workflow agent do not need identical red-team depth, even if they share the same base model.

Best practice is evolving, but current guidance suggests the highest-risk cases are systems with persistent memory, tool execution, cross-agent messaging, or access to secrets and production APIs. Those environments need scenario-based testing that includes privilege boundaries and failure propagation. Model-only tests may still be useful as a baseline, but they are not sufficient evidence of operational safety.

One common edge case is human-in-the-loop review. Teams sometimes assume an approval step neutralises system risk, but it can fail if the reviewer sees only a summarised output while the malicious action is already encoded in hidden tool output or queued automation. Another edge case is multi-agent routing, where one agent sanitises content and another later executes it. If trust assumptions are not explicit, the pipeline can become the attack surface.

For teams formalising controls, the most important rule is simple: test the exact chain of trust that production uses, not a simplified model sandbox. Anything less can leave a system that appears resilient in isolation but fails once integrated.

Related resources from NHI Mgmt Group

• What breaks when teams only monitor individual APIs instead of system behaviour?
• What breaks when teams monitor only transactions and system health instead of agent behavior?
• What breaks when teams keep rotating secrets instead of changing the access model?
• What breaks when security teams trust model confidence instead of evidence?