How do security teams balance pre-deployment testing and runtime validation for AI systems?

Why This Matters for Security Teams

AI systems create two distinct risk windows: the build phase, where testing can surface obvious jailbreaks, prompt injection paths, and data leakage, and the live phase, where users, tools, and upstream data keep changing. Security teams that rely on pre-deployment review alone are usually testing a static snapshot, not the actual production behaviour of a model connected to business data or an execution path. That is why runtime validation belongs in the control stack, not as an optional hardening layer.

This matters even more when AI can act on behalf of a user, because the system is no longer just generating text. It may be calling APIs, retrieving sensitive records, or triggering workflows, which makes the validation problem closer to access control than content moderation. The NIST Cybersecurity Framework 2.0 is useful here because it reinforces that governance, protection, and monitoring have to work together rather than sequentially.

NHI Management Group’s research on The State of Non-Human Identity Security shows how quickly weak identity controls can become an operational problem once machine identities are in play. In practice, many security teams encounter unsafe AI behaviour only after a production prompt, tool call, or secret exposure has already happened, rather than through intentional pre-launch testing.

How It Works in Practice

The strongest pattern is a layered one. Pre-deployment testing should be used to break the system before users do, while runtime validation should assume that some unsafe inputs, outputs, or tool calls will still get through. That combination is especially important when AI systems use non-human identities, because the control point is not just the model prompt. It is the identity, entitlement, and execution path behind the model.

Before launch, teams typically test for prompt injection, prompt leakage, tool abuse, data exfiltration, and policy bypass. That work is best paired with red-team style adversarial testing and threat modeling aligned to OWASP Top 10 for Large Language Model Applications and NIST AI Risk Management Framework guidance. The goal is to discover failure modes early, not to prove the model is “safe” in the abstract.

At runtime, validation should focus on the exact action being attempted. That means checking whether the prompt, retrieval result, or tool request is allowed in context; whether the identity behind the request has the right scope; and whether the output contains restricted content, secrets, or unsafe instructions. For agentic systems, this often means combining policy-as-code, short-lived credentials, and request-time authorization rather than pre-approved static access.

• Use pre-deployment tests to map likely attack paths and unsafe completions.
• Use runtime validators to block disallowed tool calls, unsafe outputs, and data movement.
• Bind each agent or workflow to a workload identity, not a shared secret.
• Re-test after every model, prompt, retrieval, or tool-chain change.

The practical lesson is that runtime controls are strongest when they enforce the same policy assumptions that were tested offline. The LLMjacking research and the DeepSeek breach both show why static assumptions fail once secrets, identities, and live integrations are involved. These controls tend to break down when the AI system has broad tool access and shared credentials because the validator can stop a bad output, but it cannot undo an over-privileged action already taken.

Common Variations and Edge Cases

Tighter validation often increases latency, false positives, and operational overhead, so organisations have to balance blocking risk against user experience and delivery speed. That tradeoff is real, especially for customer-facing systems where every extra check can affect response time.

There is no universal standard for how much of the decision should happen at pre-deployment versus runtime. Current guidance suggests a practical split: use offline testing to measure baseline risk, then reserve runtime controls for high-impact actions, sensitive data paths, and any system that can call tools or write to production. For low-risk summarisation use cases, lighter runtime checks may be enough. For systems that retrieve secrets, trigger tickets, or execute code, runtime policy must be strict.

Edge cases usually appear when teams treat the model as the product instead of the workflow around it. Multi-agent systems, retrieval-augmented generation, and autonomous assistants can each change the risk profile because the risky step may happen outside the core model response. In those environments, best practice is evolving toward continuous evaluation, context-aware policy enforcement, and per-action authorization. That is also why NHI governance and AI governance increasingly overlap: once a model can act, the identity path matters as much as the content path.

Related resources from NHI Mgmt Group

• How should security teams govern AI systems that rely on business definitions at runtime?
• How should security teams evaluate AI systems that refuse to cooperate with safety testing?
• How should teams govern runtime security for AI systems and cloud workloads?
• How should security teams govern AI systems that need both evaluation and runtime control?