Why do pre-deployment tests fail to manage production AI risk?

Why This Matters for Security Teams

Pre-deployment testing gives teams confidence that a model behaved acceptably in a controlled environment, but it does not prove safety once the system is exposed to live prompts, external tools, or changing data. That gap matters because production AI risk is shaped by runtime context: user intent, retrieval quality, tool permissions, and adversarial input. Current guidance from the NIST Cybersecurity Framework 2.0 and the NIST AI Risk Management Framework points toward continuous governance, monitoring, and response rather than one-time assurance.

Security teams often misread benchmark success as operational safety. A model can pass red-team prompts, offline evaluations, and policy checks, then fail when exposed to retrieval-augmented generation, agentic workflows, or untrusted content inserted mid-session. The issue is not only model behaviour, but also the surrounding system: connectors, memory, permissions, logging, and escalation paths. In practice, many security teams encounter AI misuse only after a production workflow has already exposed sensitive data or executed an unsafe action, rather than through intentional validation.

How It Works in Practice

Managing AI risk in production means treating the model as one control point inside a larger execution chain. Effective programmes separate pre-release assurance from runtime safeguards, then define who can intervene when a request becomes unsafe. That usually includes prompt filtering, retrieval controls, output validation, tool allowlisting, human approval for sensitive actions, and telemetry that can be sent into existing detection pipelines. The NIST Cyber AI Profile is useful here because it emphasises mapping AI-specific risks into broader cyber operations.

• Constrain what the model can access, not just what it can say.
• Validate retrieved content before it reaches the model or the user.
• Log prompts, tool calls, and policy decisions for investigation and audit.
• Apply step-up approval where an agent can trigger financial, identity, or production changes.

This matters because many failures are orchestration failures, not model failures. A safe model can still be coerced into unsafe action if it has overbroad tool permissions or if retrieved content is trusted blindly. Runtime controls also need to account for drift: the same workflow can become risky after a connector change, a prompt update, or a new data source. ISO/IEC 42001:2023 AI Management System Standard supports the governance side of that problem by formalising accountability, lifecycle oversight, and continual improvement.

These controls tend to break down when AI is embedded in high-volume workflows with weak ownership, because incidents then cross product, security, and operations boundaries faster than review processes can respond.

Common Variations and Edge Cases

Tighter runtime control often increases latency, friction, and support overhead, requiring organisations to balance safety against user experience and operational throughput. That tradeoff is real, especially where AI is used for customer support, code generation, or internal automation. Best practice is evolving on how much human review is necessary for different classes of action, and there is no universal standard for this yet. The right answer depends on whether the model only drafts content, or whether it can act through tools, identities, or funds.

Edge cases appear when AI systems interact with identity or privileged workflows. If an agent can request credentials, create tickets, approve access, or change cloud resources, the risk profile shifts from content safety to operational control. In those cases, runtime policy enforcement should be aligned with privilege boundaries, and organisations should treat the agent like any other powerful service account. This is where NHI governance, PAM, and Zero Standing Privilege become relevant even if the original question is framed as “AI testing.”

Pre-deployment tests also struggle with multimodal or highly dynamic systems, where inputs can include documents, web content, images, or tool outputs that are not stable enough to fully enumerate in advance. The practical answer is not to abandon testing, but to supplement it with continuous evaluation, incident response, and post-deployment monitoring that reflects actual usage patterns. When runtime conditions change faster than validation assumptions, offline tests stop being predictive.

Related resources from NHI Mgmt Group

• Why do build-time AI tests fail to fully reduce production risk?
• Why do post-deployment controls often fail to catch AI model risk early?
• How should security teams limit the risk from AI agents that have access to production systems?
• Why do agentic AI systems make NHI risk harder to manage?