How should security teams govern AI systems that need both evaluation and runtime control?

Use evaluation to validate quality before deployment, but require runtime controls to block unsafe prompts, outputs, and tool calls once the system is live. That means separating test coverage from policy enforcement, assigning ownership for each layer, and retaining evidence of both. Without runtime control, evaluation only tells you what happened after risk already escaped.

Why This Matters for Security Teams

AI systems that require both evaluation and runtime control create a governance gap if teams treat testing as a one-time gate. Evaluation helps measure model quality, safety, and failure modes before release, but it does not stop harmful behavior once the system is connected to users, data, and tools. Runtime control is the operational layer that enforces policy in production, including prompt filtering, output checks, tool permissioning, and escalation paths for unsafe behavior.

This distinction matters because AI risk is not static. A model can pass a test set and still fail under novel prompts, retrieval inputs, or agentic tool use. Current guidance suggests that organisations should govern AI systems through both pre-deployment assurance and live control, rather than assuming one replaces the other. That approach aligns with the NIST Cybersecurity Framework 2.0 emphasis on continuous risk management and operational oversight.

The practical mistake is to let model owners own the test plan while platform teams own production safeguards without a shared policy model. In practice, many security teams encounter unsafe AI behavior only after production usage expands faster than their control boundaries were designed to handle.

How It Works in Practice

Effective governance starts by separating three layers: evaluation, policy definition, and runtime enforcement. Evaluation is where teams test hallucination rates, jailbreak resistance, data leakage risk, and tool-use behavior against curated scenarios. Policy definition turns those findings into operational rules, such as blocked content categories, approved tool scopes, human approval thresholds, and logging requirements. Runtime enforcement then applies those rules at inference time and during agent execution.

That enforcement layer should be explicit and auditable. For example, prompt controls can inspect user input before it reaches the model, output controls can detect policy violations before the response is returned, and tool controls can restrict what an AI agent is allowed to call, modify, or exfiltrate. Where retrieval-augmented generation is used, security teams should also govern source provenance and document trust so that the model is not answering from untrusted material. NIST guidance on AI risk management and the NIST AI Risk Management Framework both support this layered approach.

A workable operating model usually includes:

• pre-release evaluation criteria tied to business use cases and abuse cases
• policy owners who define what content, actions, and tool calls are permitted
• runtime safeguards that enforce those rules consistently across applications
• telemetry and evidence retention for prompts, outputs, decisions, and overrides
• incident response playbooks for model misbehavior, prompt injection, and tool misuse

For agentic systems, control of execution authority is especially important. If an AI agent can send email, query records, or trigger workflows, then the security model must treat those actions like privileged operations, not mere content generation. MITRE’s ATT&CK framework remains useful for mapping abuse patterns to operational detections, while the MITRE ATLAS knowledge base helps teams think about adversarial manipulation of AI systems. These controls tend to break down when teams embed AI into legacy workflows with no central policy enforcement point because prompts, tools, and audit data become fragmented across services.

Common Variations and Edge Cases

Tighter runtime control often increases latency, engineering overhead, and false positives, requiring organisations to balance safety against usability and throughput. That tradeoff is especially visible in customer-facing assistants, internal copilots, and autonomous agents, where excessive blocking can make the system impractical and overly permissive settings can make it unsafe.

Best practice is evolving for multi-model and multi-tenant environments, because there is no universal standard for how much control should sit in the application, the gateway, or the model-serving layer. Some organisations centralise policy at the orchestration layer, while others push guardrails into each application for lower latency and clearer business ownership. The right answer depends on how sensitive the tasks are, how often the model changes, and how much tool access the system has.

Edge cases also arise when evaluation and runtime control are owned by different teams. That separation can work, but only if the evidence chain stays intact: test results should map to production controls, and runtime incidents should feed back into evaluation scenarios. Where AI systems interact with identity, secrets, or privileged workflows, the governance model should become stricter, not looser, because a single prompt or tool-call failure can become a broader security incident. For practical control mapping, teams often align with OWASP guidance for LLM applications and emerging agentic security practices.

Related resources from NHI Mgmt Group

• How should security teams govern AI systems that rely on business definitions at runtime?
• How should teams govern runtime security for AI systems and cloud workloads?
• How should security teams govern AI agents that can access enterprise systems?
• How should security teams govern AI agents that can take runtime response actions?