How should security teams validate LLMs and agents under SR 11-7?

Why This Matters for Security Teams

SR 11-7 was written for model risk governance, but its core expectation applies cleanly to LLMs and agents: validation must be independent, evidence-based, and repeated as the system changes. For security teams, the practical issue is that generative systems can fail in ways traditional model testing does not capture, including prompt injection, hallucinated actions, tool misuse, and unsafe escalation paths. Guidance from the NIST AI Risk Management Framework reinforces that AI assurance is not a one-time gate, but a lifecycle discipline tied to governance, measurement, and ongoing monitoring.

The validation question matters most when an LLM is not just producing text, but shaping decisions, generating code, or driving workflows through tools and connectors. At that point, the risk is no longer limited to model quality. It extends to authorization boundaries, data leakage, downstream business impact, and the ability to explain why a control failed after the fact. Security teams often assume a strong red-team exercise or a pre-launch benchmark is enough, but SR 11-7 style validation is about demonstrating the system remains within acceptable risk tolerances under operational conditions. In practice, many security teams encounter AI model risk only after a tool has already been invoked with the wrong state, rather than through intentional pre-production challenge testing.

How It Works in Practice

Validating LLMs and agents under SR 11-7 should follow a documented control process that covers design, test evidence, production monitoring, and revalidation triggers. The test plan should be specific to the system’s use case, data sources, and action scope. For LLMs, that means checking output consistency, refusal behaviour, bias boundaries, prompt sensitivity, and resilience to adversarial inputs. For agents, validation must extend to tool selection, action ordering, approval logic, and whether the system can be constrained from making irreversible external changes.

Security teams should treat validation artifacts as audit-ready evidence, not informal reassurance. That usually includes:

• Defined acceptance thresholds for accuracy, toxicity, hallucination, and unsafe completion rates.
• Adversarial test sets that include prompt injection, data exfiltration attempts, and jailbreak patterns informed by the MITRE ATLAS adversarial AI threat matrix.
• Agent-specific checks for tool permissions, step sequencing, and human approval gates before high-impact actions.
• Monitoring for drift in model behaviour, retrieval quality, policy adherence, and connector trustworthiness.
• Revalidation after model updates, prompt changes, tool additions, or shifts in the underlying data population.

Where available, teams should also align test scenarios to the OWASP Agentic AI Top 10 and the NIST AI 600-1 Generative AI Profile, because these resources help translate abstract risk language into testable failure modes. These controls tend to break down when the agent is granted broad API access across multiple systems, because the validation environment no longer mirrors the real trust boundaries or the blast radius of a bad decision.

Common Variations and Edge Cases

Tighter validation often increases delivery overhead, requiring organisations to balance release speed against the need for defensible assurance. That tradeoff becomes sharper with agentic systems, where even a low-probability failure can trigger a high-impact state change. Current guidance suggests that teams should not rely on a single universal test suite for all LLMs and agents, because the acceptable risk threshold depends on whether the system drafts content, advises operators, or executes actions.

Edge cases usually appear in environments with retrieval-augmented generation, third-party tools, or delegated credentials. In those settings, the model may be safe in isolation but unsafe in context, especially if retrieval sources are stale, tool schemas are too permissive, or the agent can chain benign actions into an unintended outcome. The CSA MAESTRO agentic AI threat modeling framework is useful here because it reinforces that system-level trust, not just model quality, must be validated.

There is no universal standard for agent validation thresholds yet, so organisations should document their own risk appetite and explain why specific failure rates are acceptable for a given use case. That is especially important where agents interact with regulated data, privileged systems, or external parties. In those cases, validation should include rollback plans, incident triggers, and a clear line between safe recommendation and unsafe execution. If the system can commit an action that cannot be easily reversed, the validation bar should be materially higher than for a read-only assistant.

Related resources from NHI Mgmt Group

• How should security teams govern AI agents under SOC 2?
• How should security teams manage permissions for AI agents?
• How should security teams authenticate AI agents in enterprise environments?
• How should security teams govern AI agents that use OAuth access?