Who is accountable when an AI model changes and the inventory is wrong?

Why This Matters for Security Teams

When an AI model changes and the inventory lags behind, accountability gaps quickly turn into control gaps. The issue is not only who approved the model, but who owned the evidence that the inventory remained accurate after deployment, retraining, vendor updates, or toolchain changes. That matters for auditability, incident response, and change governance, especially where the model can trigger downstream actions or access other systems.

Security teams often discover that “ownership” was assumed rather than explicitly assigned. A business owner may understand the use case, a technical team may understand the deployment, and compliance may review the paperwork, but none of them can prove who was accountable for keeping the record current. Current guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls reinforces the need for defined control ownership and change accountability, which is exactly where inaccurate inventories undermine governance.

In practice, many security teams encounter the mismatch only after a model change has already shipped, rather than through intentional inventory reconciliation.

How It Works in Practice

Accountability should be treated as a chain of custody for the model record. The named business owner is responsible for the operational purpose and risk acceptance. The technical approver is responsible for validating the model version, deployment state, dependencies, and any change in capability or data flow. The compliance or control owner is responsible for ensuring the evidence exists to show that the inventory matches reality at the time of review.

That chain only works if the organisation defines what counts as a material change. For example, a new prompt template, a retrained model, a switched API endpoint, or a changed tool permission may each require inventory updates even if the user-facing service name stays the same. Best practice is evolving here, but the practical rule is simple: if the model can behave differently, its inventory record must reflect that difference.

• Assign one accountable owner for the record, not just for the system.
• Link each model entry to version, deployment environment, and approval date.
• Record vendor dependencies, external APIs, and tool access as part of the asset record.
• Require evidence of review after retraining, fine-tuning, or configuration changes.
• Correlate inventory updates with change tickets and logging so the timeline is defensible.

This is closely aligned with control expectations in NIST AI Risk Management Framework, especially around governance and traceability, and with OWASP Top 10 for LLM Applications where untracked model behaviour can create downstream security and integrity issues.

These controls tend to break down when models are embedded inside platform teams, SaaS features, or CI/CD pipelines because ownership becomes distributed across teams that do not share a single change record.

Common Variations and Edge Cases

Tighter inventory governance often increases operational overhead, requiring organisations to balance fast model iteration against evidence quality and approval discipline. That tradeoff becomes more visible in environments where models are updated frequently, where third-party providers silently refresh model endpoints, or where agentic systems can alter their own tool use.

There is no universal standard for this yet on how often model inventories must be reconciled, but current guidance suggests using change-triggered review rather than relying only on periodic audits. In highly regulated settings, a missed inventory update can become a control failure even when the underlying model change was technically low risk.

Edge cases also matter. If a model is retired but still reachable through an old API key, the inventory may say “inactive” while the environment still allows execution. If multiple teams reuse the same foundation model through different wrappers, the owner of the wrapper may not be the owner of the underlying model risk. Where agentic AI is involved, accountability should extend to the tool permissions and service accounts the agent uses, not just the model artifact itself.

For governance and control mapping, teams should also align the inventory record to OWASP Agentic AI Top 10 and, where autonomous behaviour creates exposure, the adversarial risk lens in MITRE ATLAS.

Related resources from NHI Mgmt Group

• Who is accountable for AI policy violations when the judge model is wrong?
• Who should be accountable for AI-assisted deliverables when the model is wrong?
• Who is accountable when production data changes an AI control model?
• Who is accountable when an assistant applies a remediation that changes cloud configuration?