Both, because a model that internalises vulnerable patterns is a governance issue and an operational exposure. Validation should sit before production approval, alongside ongoing monitoring after deployment. Organisations that let models touch security-sensitive code or incident workflows without this control are effectively accepting unknown internal behaviour.
Why model validation belongs to both governance and operations
Model validation is a governance control because it determines whether a model is fit for approved use, within risk appetite, and acceptable for the intended workflow. It is also an operational control because models change behaviour over time, drift after deployment, and can create immediate exposure if they influence decisions, code, or incident response. Treating validation as a one-time sign-off leaves a gap between approval and reality.
For governance teams, validation should answer whether the model is allowed into a specific use case, with clear ownership, documented testing criteria, and explicit acceptance of residual risk. For security teams, the same control should be translated into observable checks on outputs, prompts, tool use, and failure modes so that unsafe behaviour is detected when the model is live. The control only works when both views exist together.
This is especially important for high-impact uses such as security-sensitive code review, alert triage, incident drafting, or automated remediation. In those settings, validation is not about proving the model is generally useful, it is about proving it will not introduce unsafe recommendations, excessive confidence, or brittle behaviour in the exact workflow where it will be relied on.
What effective validation should cover before and after deployment
Pre-deployment validation should focus on the intended use case, not the model in isolation. That means testing the model against realistic prompts, known edge cases, disallowed actions, and the specific data it will see in production. It should also check whether the model’s outputs remain bounded when users try to push it beyond its approved role.
Post-deployment validation should monitor for behavioural drift, new failure patterns, and changes in downstream impact. A model that looked acceptable in testing can still become unsafe if the surrounding tooling changes, the prompt format changes, or users begin to rely on it in ways that were not reviewed. Ongoing validation is therefore part of operational assurance, not just release management.
Validation is strongest when it is tied to the AI Governance Guide for approval, ownership, and risk-tiered review, and to the AI Security Maturity Model for understanding how validation fits into a broader control programme. When the model has security-sensitive output or tooling, validation also needs to be informed by the threat paths described in the Agentic AI Security Guide, especially where misuse of tools or identity-sensitive actions could become operationally dangerous.
Where organisations go wrong when they split governance from security operations
The most common failure is treating model approval as a paperwork exercise and operational monitoring as someone else’s problem. That creates a blind spot: the governance team may approve the model because the demo looked safe, while the operations team only notices unsafe behaviour after users have already trusted it in production.
Another failure is validating only for accuracy or helpfulness while ignoring control failure. A model can be “good enough” on benchmark tasks and still be dangerous if it confidently suggests insecure changes, invents incident steps, or ignores guardrails when the prompt context becomes adversarial. That is why validation must include misuse scenarios, not just expected happy-path tasks.
Organisations that need a control-oriented testing pattern should borrow the mindset of Breach and Attack Simulation (BAS) Guide and adapt it to AI behaviour: validate repeatedly, under realistic pressure, and against the exact decisions the model is allowed to influence. For model security itself, the AI Model Security Guide is the right reference point when the concern is extraction, inversion, adversarial examples, or inference abuse rather than governance alone.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST AI RMF, NIST AI 600-1 and NIST CSF 2.0 set the technical controls, while ISO/IEC 42001:2023 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| ISO/IEC 42001:2023 | 8.1 — Operational Planning and Control | Model validation must be controlled before release and during operation. |
| Recommendation — Define operational validation criteria and keep them under controlled change management. | ||
| NIST AI RMF | MAP — Measure | Validation is the measurable assurance activity for AI behaviour and risk. |
| Recommendation — Measure model behaviour against documented risk and performance criteria before approval and after deployment. | ||
| NIST AI 600-1 | GV-3 — Governance and Risk Management | GenAI validation supports governance decisions about acceptable use and oversight. |
| Recommendation — Require governance review of validation evidence before authorising high-impact GenAI use. | ||
| NIST CSF 2.0 | GV.RM-01 — Risk Management Strategy Established | Validation supports a defined risk appetite for AI use in security-sensitive workflows. |
| DE.CM-09 — Configuration Change Monitoring | Post-deployment validation depends on monitoring behaviour changes and drift. | |
| Recommendation — Set validation thresholds that align AI use with organisational risk tolerance. Monitor AI outputs and configuration changes for behaviour drift after deployment. | ||
Practitioner Guidance
What to prioritise: Separate “approval to use” from “continuous assurance after use,” but keep one accountable owner for both. If no team owns the full validation loop, the control will fragment into policy without monitoring or monitoring without decision authority.
What to verify: Check that validation criteria are tied to the exact workflow, not generic model quality. If the model can influence code, security tickets, or incident decisions, the test set should include unsafe suggestions, refusal failures, hallucinated certainty, and prompt conditions that are likely to occur in production.
What good looks like: The organisation can show that the model was approved for a bounded use case, is being monitored for drift or unsafe output patterns, and has a clear escalation path when behaviour changes. That is materially different from a one-time benchmark report filed at launch.
Practitioner takeaway: Treat validation as a control that decides both whether the model may be used and whether it remains safe to keep using, because the risk lives in deployment behaviour, not just in initial test results.
Related resources from NHI Mgmt Group
- Should organisations treat model retention policies as part of security governance?
- Should organisations treat AI observability as part of IAM and governance or as a separate security tool?
- Should organisations treat search results and email as part of AI security governance?
- Should organisations treat runtime monitoring as part of AI governance or as a separate security task?