AI model governance in production needs stronger control and audit

At a glance

What this is: This is a governance analysis of AI model governance in production, focused on the gap between model registries and what is actually live.

Why it matters: It matters because identity, accountability, and control boundaries around AI systems now affect both human oversight and non-human operational risk across the enterprise.

By the numbers:

• 72% of organisations have experienced or suspect they have experienced a breach of non-human identities.

👉 Read Collibra's analysis of AI model governance in production

Context

AI model governance is the operational discipline that tracks, audits and controls individual models once they are in production. The problem Collibra describes is simple: organisations often know what exists in the registry, but not what is actively influencing decisions in live systems.

That gap matters to IAM and governance teams because it mirrors the same control problem seen in non-human identity programmes. Ownership is assumed, monitoring is inconsistent, and retirement is too often informal, which turns live AI assets into unmanaged production identities.

Key questions

Q: How should security teams govern AI models that are already in production?

A: They should treat production models as controlled assets with named ownership, documented purpose, approval history, monitoring thresholds and a clear retirement path. The key is to reconcile the registry with live usage so governance reflects what is actually influencing decisions. Without that reconciliation, teams are auditing records instead of systems.

Q: What breaks when AI model governance stops at the registry?

A: The organisation loses visibility into what is actually running, who owns it and whether its behaviour still matches the approved state. That creates audit gaps, weak incident response and unmanaged risk when deprecated or drifting models remain live in business processes.

Q: How do teams know whether model governance is working?

A: They can prove it when every live model is mapped to an owner, a model card, lineage data, monitoring thresholds and a retirement decision trail. If any of those elements are missing, governance is partial and the organisation cannot reliably defend its control state.

Q: Who is accountable when a production model drifts below approved thresholds?

A: The named model owner is accountable for the review, escalation and decision path, even if engineering or data science detects the issue first. Governance fails when drift is treated as a technical alert without a business owner attached to the next decision.

Technical breakdown

Production blind spots in AI model inventories

A model inventory is only useful if it matches what is actually deployed. The production blind spot appears when models are registered, approved or documented, but not continuously reconciled against live systems that score transactions, generate outputs or inform decisions. In practice, this creates a control gap between governance records and runtime reality. If a model is deprecated in name but still called by downstream applications, the organisation is governing paperwork, not behaviour. The same issue appears when ownership follows the original builder instead of a named accountable function. Practical implication: reconcile registries with production dependencies on a fixed cadence.

Practical implication: Reconcile the model registry against live production usage on a fixed cadence.

Model cards, lineage and the audit trail problem

Model cards are the control artefact that turns model metadata into an audit trail. They should capture purpose, training data, risk tier, ownership, approval history and change log. Lineage extends that evidence by linking training data, model versions and downstream consumers so teams can see what changed and who was affected. Without lineage, a team may know a model was approved, but not whether its training inputs later became sensitive, biased or obsolete. That makes incident response and compliance evidence brittle. Practical implication: connect model metadata to lineage so governance evidence follows the model through its full lifecycle.

Practical implication: Connect model metadata to lineage so governance evidence follows the model through its full lifecycle.

Drift detection as a governance control

Drift is not only a machine learning quality problem. When a model’s performance falls below the threshold under which it was approved, the approved control state no longer matches operational reality. That matters because governance decisions are made on the assumption that the reviewed model is the one still running. Continuous monitoring, threshold-based alerts and documented escalation therefore belong in the governance layer, not only in engineering telemetry. If drift is only visible to data scientists, the organisation has no effective governance response. Practical implication: treat drift thresholds as policy triggers that require review, restriction or retirement decisions.

Practical implication: Treat drift thresholds as policy triggers that require review, restriction or retirement decisions.

NHI Mgmt Group analysis

Production model inventories are a governance control, not a documentation exercise. The article exposes a familiar failure mode: teams believe a registry is a control, but the registry only helps if it stays aligned to what is live. This is structurally similar to unmanaged non-human identities, where the record exists but the operational reality has drifted. Practical implication: treat production reconciliation as part of governance, not an admin task.

Model ownership that depends on the original builder is a brittle accountability model. Collibra’s example of ownership being implicit until someone leaves shows how easily accountability decays when it is person-based rather than function-based. That same pattern appears in non-human identity lifecycle management when access persists after team changes or departures. Practical implication: require durable ownership for every live model, with explicit reassignment on role change.

Drift creates an approval mismatch, not just a performance issue. A model approved at one accuracy level and later operating well below that threshold is no longer the same governed system. The governance assumption that an approval remains valid until the next formal review fails when the operating context shifts materially between reviews. Practical implication: make drift a formal governance trigger, not an engineering note.

Model governance now sits on the same control plane as identity lifecycle discipline. The article’s lifecycle framing, intake through retirement, is the right mental model because AI models behave like governed runtime assets, not static software releases. The strongest organisations will align this with NIST AI RMF and NIST Cybersecurity Framework style accountability, while keeping the operational evidence tied to the live asset. Practical implication: build model governance as a lifecycle programme with named accountability and revocation paths.

Runtime evidence is the named concept this article reinforces: production blind spot. The blind spot is not that organisations lack AI ambition. It is that their governance evidence trail stops at deployment and never proves what remained active, monitored and retired in practice. That leaves audit, incident response and compliance all working from incomplete facts. Practical implication: close the blind spot by making live-state evidence a first-class control.

From our research:

• 85% of organisations lack full visibility into third-party vendors connected via OAuth apps, according to The State of Non-Human Identity Security.
• A second finding from The 2024 ESG Report: Managing Non-Human Identities shows that 72% of organisations have experienced or suspect they have experienced a breach of non-human identities.
• For a broader governance lens, read NHI Lifecycle Management Guide for how visibility, ownership and retirement should work across governed identities.

What this signals

Production blind spot: this is the pattern practitioners should now watch for across AI estates, where the registry says one thing and the runtime estate says another. With 72% of organisations reporting or suspecting NHI breaches, per The 2024 ESG Report: Managing Non-Human Identities, unmanaged live-state evidence is becoming a governance liability, not just an operations problem.

Teams should expect AI model governance to converge with identity lifecycle discipline, especially around ownership, monitoring and retirement. The practical shift is toward verifiable control evidence, not policy statements, and NIST Cybersecurity Framework 2.0 remains a useful reference point for that control mapping.

For practitioners

• Reconcile live models against registry records Compare the model registry, deployment logs and application dependencies to find models that are still active but no longer documented or owned.
• Assign durable ownership for every production model Name one accountable business or technical owner per model and force ownership changes when teams or roles change.
• Tie drift thresholds to governance decisions Define the performance threshold that triggers review, restriction, retraining or retirement, and route those alerts to the accountable owner.
• Connect model cards to lineage evidence Ensure each model card links training inputs, validation data, approval history and downstream consumers so auditors can trace impact quickly.
• Formally retire models that are no longer used Remove silent, deprecated or abandoned models from service through documented decommissioning rather than assuming inactivity means retirement.

Key takeaways

• AI model governance fails when organisations cannot prove which models are live, who owns them and how they are monitored.
• The scale of the problem is operational, not theoretical, because a registry without runtime reconciliation leaves blind spots in audit and response.
• Practitioners should align model cards, lineage, drift thresholds and retirement workflows so governance evidence matches the production estate.

Key terms

• Production Blind Spot: The gap between what governance records say is deployed and what is actually influencing decisions in live systems. In model governance, it appears when teams track registry entries but cannot prove ownership, monitoring or retirement of the real production asset.
• Model Card: A structured record for one AI model that captures purpose, data sources, risk tier, ownership, approval history and known limitations. It is the primary evidence artefact that lets auditors and operators understand what a model is meant to do and who is responsible for it.
• Model Drift: The degradation of a model’s behaviour or performance as real-world inputs diverge from the conditions under which it was trained and approved. In governance terms, drift matters because it can invalidate the assumptions behind an earlier approval and trigger a review or retirement decision.
• Lineage: The traceable relationship between data sources, model artefacts and downstream outputs or decisions. Lineage lets practitioners see how training data flowed into a model and which systems or business actions were affected, which is essential for auditability and impact analysis.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or governance in your organisation, it is worth exploring.

This post draws on content published by Collibra: AI model governance in production. Read the original.