How should security teams govern AI systems that need both evaluation and runtime control?

Why evaluation and runtime control must be governed separately

Security teams should treat evaluation and runtime control as two different governance layers. Evaluation answers whether the system is good enough to ship, while runtime control answers whether the system can be safely used when inputs, outputs, and tool actions become unpredictable. If you blur the two, you create a gap between test-time confidence and live-time exposure.

That separation matters because evaluation is retrospective and bounded, but runtime enforcement is continuous and adversarial. A system can look acceptable in a test set and still generate unsafe content, follow a manipulated prompt, or call a tool outside policy once it is deployed. Governance should therefore define which layer validates, which layer enforces, and which layer owns each decision.

For teams building or buying controls, the useful question is not whether the model passed evaluation once. It is whether the production path can still stop a bad prompt, constrain a risky output, or deny an unapproved action after the system is connected to real users, data, and tools. The AI Security Platform Buyer's Guide is useful here because it frames runtime guardrails and vendor evaluation as separate buying and assurance problems.

What good governance looks like across test, policy, and enforcement

Good governance starts by defining the control boundary. Evaluation should measure accuracy, refusal behaviour, prompt-injection resistance, tool-use correctness, and other acceptance criteria before release. Runtime controls should then enforce the policy independently, so a passing evaluation does not become a blanket exception for live behaviour. That is especially important when the system can take action, not just generate text.

Ownership should also be split cleanly. The team running evaluation needs criteria, test cases, and release gates. The team operating runtime controls needs policy definitions, escalation paths, logging, and rollback authority. If one team owns both without a clear handoff, failures often get rationalised as “known model behaviour” rather than treated as a live control defect.

From a program standpoint, the architecture should make evidence easy to produce. Keep the evaluation results, the live policy set, and the runtime enforcement logs together so reviewers can see not only that the system was tested, but that the same class of risk was still controlled after deployment. NHIMG’s AI Security Architecture Reference Guide is a good navigation point for layered controls spanning application, agent, identity, and monitoring layers.

For organisations trying to formalise the management system side, the ISO/IEC 42001 Guide is relevant because it helps structure accountability, impact assessment, and evidence retention around AI governance rather than around model quality alone.

How to keep unsafe prompts, outputs, and tool calls under control in production

Runtime control should be designed to intercept the exact failure mode, not just observe it. Unsafe prompts may need input filtering, context scoping, or policy checks before the model sees them. Unsafe outputs may need moderation, classification, or hard blocking before a user receives them. Unsafe tool calls need explicit authorization checks, scoped permissions, and logging so that actions are bounded by policy rather than by the model’s confidence.

The practical mistake is to rely on evaluation artefacts as if they were enforcement. A test proves the system can behave safely under controlled conditions; it does not stop a malicious prompt from exploiting a gap later. In live service, the safest designs assume that some prompts will be hostile, some outputs will be wrong, and some tool requests will be inappropriate unless the runtime layer actively intervenes.

The most useful operating model is layered: one layer tests whether the system should be approved, and another layer decides what the system may do right now. That approach also makes incident response clearer, because a policy miss in production becomes a control failure to investigate, not just a quality issue to retest.

Teams that need a stronger external governance anchor can use the NIST AI 600-1 GenAI Profile for pre-deployment validation, and the NIST AI Risk Management Framework for broader governance of trustworthy AI lifecycle decisions.

Risk and Threat Considerations

If evaluation and runtime control are merged conceptually, the main risk is false confidence. A system may score well in testing and still leak unsafe content, accept manipulated instructions, or invoke a tool in a way that creates real operational harm once deployed. The threat is not limited to model failure, because adversaries can target the live policy boundary itself.

Failure mechanism: evaluation only measures a bounded test environment, while runtime exposure depends on real prompts, real users, live context, and actual tool permissions. If the production layer has no independent enforcement, the first unsafe action may occur before anyone notices the test results are no longer representative.

Impact: unsafe outputs can reach users, privileged tool calls can execute, and the organisation may discover the weakness only after data exposure, workflow abuse, or a customer-impacting incident. That is why runtime control must be treated as a separate security control, not as a continuation of testing.

Practitioner Guidance

What to verify: confirm that evaluation gates and runtime policy checks are independently documented, separately owned, and separately testable. If the same control is doing both jobs, you do not yet have governance separation.

Decision rule: if a control prevents release only in pre-production, treat it as evaluation; if it can still stop unsafe behaviour after deployment, treat it as runtime enforcement. Production approval should never depend on a control that disappears at go-live.

Practitioner takeaway: the standard for AI governance is not “did we test it well?”, it is “can we still constrain it safely when the system is live and interacting with real users, data, and tools?”

Related resources from NHI Mgmt Group

• Govern
• How should security teams govern AI systems that rely on business definitions at runtime?
• How should teams govern runtime security for AI systems and cloud workloads?
• How should security teams control AI evaluation environments so autonomous agents cannot escape into production systems?