Why do AI governance programmes need ongoing evaluation when models are already approved for use?

Why This Matters for Security Teams

AI approvals are often treated as a finish line, but in practice they are only a snapshot of risk at a point in time. Once a model is connected to live data, user prompts, plugins, or downstream workflows, its behaviour can shift in ways that procurement reviews do not capture. Ongoing evaluation is essential to verify that controls still work, outputs remain within acceptable boundaries, and business teams are not relying on stale assumptions.

This matters because AI risk is not limited to the model itself. It also includes prompt handling, retrieval sources, vendor updates, and how people interpret the outputs. The NIST AI Risk Management Framework treats AI risk as something to be governed throughout the lifecycle, not only at approval. That lifecycle view aligns with operational reality: controls that looked adequate in testing can become weak after integration, retraining, or model version changes.

Security and governance teams also need evidence for audit, incident response, and regulatory accountability. If evaluation stops after go-live, there is no reliable way to show whether the model still behaves as intended or whether a third-party update introduced new exposure. In practice, many security teams encounter AI risk only after a harmful output, business complaint, or policy breach has already occurred, rather than through intentional monitoring.

How It Works in Practice

Ongoing evaluation means building review into the operating model, not treating it as a one-time validation task. The baseline should define what “safe and acceptable” means for the use case, then test those expectations repeatedly against real prompts, live retrieval content, and changed conditions. For generative systems, the NIST AI 600-1 Generative AI Profile is useful because it emphasises evaluation across the full GenAI lifecycle, including output quality, misuse resistance, and governance.

Practitioners typically monitor several dimensions:

• Output quality, including factuality, relevance, and internal consistency.
• Safety behaviour, including prompt injection resistance and refusal handling.
• Data integrity, especially the trustworthiness of retrieved or fine-tuning data.
• Change management, including vendor model updates and configuration drift.
• Logging and review, so exceptions can be traced back to inputs and model versions.

For broader organisational control, the NIST Cybersecurity Framework 2.0 helps connect AI monitoring to governance, risk management, and continuous improvement. Organisations with formal AI management programmes often also map their controls to ISO/IEC 42001:2023 AI Management System Standard to show repeatable oversight, ownership, and corrective action.

Where models support threat detection, triage, or response workflows, evaluation should also account for adversarial behaviour and false confidence in machine recommendations. The NIST Cyber AI Profile (IR 8596) is particularly relevant for checking whether cyber-focused AI remains dependable under real attack conditions. These controls tend to break down when teams allow rapid vendor-side model updates without revalidation because the approved baseline no longer matches production behaviour.

Common Variations and Edge Cases

Tighter evaluation often increases operational overhead, requiring organisations to balance stronger assurance against speed and resource constraints. That tradeoff is especially visible for high-volume GenAI services, where manual review does not scale and sampling strategies must be carefully designed.

Best practice is evolving for systems that use agentic workflows, RAG pipelines, or externally updated foundation models. There is no universal standard for how often every model must be retested, but current guidance suggests frequency should reflect business impact, model volatility, and exposure to untrusted inputs. A low-risk internal drafting assistant may warrant lighter review than a customer-facing decision-support tool.

Edge cases also arise when governance teams assume approval transfers across contexts. A model approved for summarisation is not automatically approved for fraud screening, hiring support, or safety-sensitive advice. Similarly, a vendor’s assurance pack may not cover local prompt templates, retrieval sources, or policy overlays that materially change the risk profile. In regulated environments, the EU AI Act reinforces this lifecycle view by expecting appropriate risk management, documentation, and post-deployment oversight for covered systems.

For teams managing AI that touches security operations, ongoing evaluation should be aligned with incident response and change control, not isolated in model governance meetings. The strongest programmes treat drift, abuse, and vendor updates as recurring control events, not exceptions. Where oversight breaks down, it is usually because ownership is split between procurement, data science, and operations, leaving no single team accountable for revalidation after change.

Related resources from NHI Mgmt Group

• What fails when AI models can use credentials during evaluation?
• How should security teams use LLM tracing in AI governance programmes?
• How should organisations use AI agents in access reviews without losing governance control?
• How should organisations use AI in IAM without weakening governance?