How should organisations approve AI models for real-world use?

Why This Matters for Security Teams

Model approval is a governance decision, not a one-time technical sign-off. A model that performs well in a lab can still fail when exposed to sensitive data, prompt injection, jailbreak attempts, or workflow-specific instructions. Security teams should treat approval as a controlled release into a defined environment, with explicit ownership, evidence, and boundaries. That approach aligns with the NIST Cybersecurity Framework 2.0, which emphasises governance, risk, and resilience rather than isolated testing.

The practical risk is not only unsafe outputs. Poorly approved models can leak regulated data, amplify bias, or create false confidence in automated decisions. They can also become a supply chain risk when the model, adapter, or retrieval layer changes after approval. For AI systems that interact with tools or agents, the approval standard must also consider whether the model can trigger actions, escalate access, or expose secrets through downstream integrations. That is where AI governance and identity governance intersect: the model may not be a human identity, but it can still need tightly scoped authority.

In practice, many security teams encounter model misuse only after the model has already been embedded into a business process, rather than through intentional release gating.

How It Works in Practice

Real-world approval starts with the use case. The organisation should document what the model is allowed to do, what data it may see, what outputs it may generate, and what actions those outputs may trigger. That scope should drive the evaluation plan, because a customer service assistant, a code-generation model, and a fraud triage model carry very different risks. For AI-specific threat modelling, current guidance from OWASP Top 10 for LLM Applications and the MITRE ATLAS knowledge base is useful for identifying prompt injection, model extraction, adversarial prompting, and data leakage scenarios.

Approval evidence should include testing against the actual workflow, not just benchmark scores. That means validating system prompts, retrieval sources, tool permissions, output filters, and escalation paths under realistic conditions. It also means checking whether the model behaves safely when the input is malformed, contradictory, malicious, or outside its training distribution. Organisations should require traceability for model provenance, training or fine-tuning data sources, versioning, and any post-training modifications. If the model is part of an agentic workflow, the approval should also cover whether human approval is required before actions are executed.

• Define the business purpose and prohibited uses before any production release.
• Test privacy, bias, hallucination, and adversarial robustness in the target workflow.
• Restrict data access by classification and validate that outputs cannot expose secrets.
• Require monitoring for drift, abuse, and change in model behaviour after deployment.
• Document rollback, kill switch, and re-approval triggers for prompt, model, or data changes.

Where high-stakes decisions are involved, organisations should also align governance with the NIST AI Risk Management Framework so the approval process covers accountability, measurement, and ongoing monitoring. These controls tend to break down when the model is wrapped in a rapidly changing RAG or agentic pipeline because the approval boundary no longer matches the live system.

Common Variations and Edge Cases

Tighter model approval often increases release overhead, requiring organisations to balance speed against assurance. That tradeoff becomes more acute when product teams want to swap models frequently or personalise outputs per user. In those environments, best practice is evolving: there is no universal standard for how often a model must be re-approved after a vendor update, adapter change, or retrieval corpus refresh. The safer approach is to define materiality thresholds that trigger review.

Edge cases include open-weight models, external API models, and embedded copilots. Open-weight models often require stronger local controls around provenance, patching, and abuse testing. External API models shift some risk to the provider, but they do not remove the obligation to assess data exposure, output handling, and contractual safeguards. Copilots used inside privileged workflows deserve extra scrutiny because a harmless-looking model can become an access path to files, tickets, or administrative actions.

For regulated use cases, approval should be documented as an operational control, not a one-off innovation exception. Where personal data, financial decisions, or identity evidence are involved, the review should be expanded to include retention, auditability, and human challenge mechanisms. If the model can influence access decisions or act on behalf of a user, the approval process should explicitly define identity boundaries and escalation rules. This matters most in environments with frequent prompt changes, shared tool access, or unmanaged retrieval sources.

Related resources from NHI Mgmt Group

• What should organisations monitor in AI workflows that use reasoning models?
• How should organisations measure trust across AI use cases, agents, and models?
• When should organisations use AI to help manage NHIs?
• How can organisations govern AI agents that use service accounts and tokens?