Who is accountable for approving machine learning models that lack explainability and monitoring?

Accountability usually sits with the product, engineering, risk, and compliance stakeholders that approve model release. In regulated environments, they need evidence that the model can be understood, tested, and governed before deployment. If explainability or monitoring is missing, approval becomes a governance decision as much as a technical one, because the organisation is accepting unknown model risk.

Why This Matters for Security Teams

Approving a machine learning model without explainability or monitoring is not just a delivery milestone, it is an explicit acceptance of model risk. That matters because opaque models can fail in ways that are hard to detect, hard to investigate, and hard to justify after the fact. Current guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls supports accountability for system integrity, auditability, and continuous oversight, all of which are weakened when the model cannot be explained or monitored effectively.

Security, risk, and product leaders often treat model approval as a technical sign-off, but the real issue is whether the organisation can defend the decision to deploy. If the model influences eligibility, prioritisation, fraud handling, or access decisions, then poor visibility becomes a governance and legal exposure as well as an operational one. This is especially true when the model is embedded in a workflow that humans trust by default, even when the output is uncertain.

In practice, many security teams encounter model risk only after a bad prediction, a complaint, or an incident review has already exposed the absence of monitoring and accountability.

How It Works in Practice

Accountability should follow the organisation’s AI governance structure, not the model owner’s preference. In most mature programmes, the person who can approve release is not acting alone. They rely on a documented review from product, engineering, security, privacy, legal, and risk functions, with each group owning a distinct part of the decision. For high-impact use cases, the approver should be able to show what the model does, what it does not do, and what controls exist to detect drift, abuse, and unsafe outputs.

Practically, that means approval should be tied to evidence. Teams usually look for model cards or equivalent documentation, testing results, baseline performance metrics, change logs, and monitoring plans. For AI systems, NIST AI Risk Management Framework is useful because it frames governance as an ongoing function rather than a one-time gate. Where a model has tool access, agentic behaviour, or dependency on retrieval systems, monitoring also needs to cover prompt injection, data leakage, and unsafe action execution.

• Define who owns approval, who advises, and who can block release.
• Require documented evidence for explainability limits, test coverage, and known failure modes.
• Set monitoring for drift, anomalies, performance degradation, and policy violations.
• Escalate unresolved risk to a named executive owner before production use.

Where possible, align the approval record to MITRE ATLAS so adversarial tactics against AI systems are part of the review, not an afterthought. These controls tend to break down when the model is deployed through a fast-moving product pipeline with no named risk owner and no production telemetry.

Common Variations and Edge Cases

Tighter approval controls often increase delivery friction, requiring organisations to balance speed against defensibility. That tradeoff is real, especially for experimental models, internal copilots, and low-risk automation where a full governance stack may be heavier than the use case demands. Best practice is evolving here, and there is no universal standard for every deployment tier.

In some environments, a model may be approved for limited use even if explainability is partial, provided there are compensating controls such as human review, constrained actions, rollback capability, and explicit user disclosures. In others, especially where the model affects regulated decisions, the approval bar should be higher and may require documented residual-risk acceptance by senior risk or compliance leadership. If the model is supplied by a third party, accountability does not disappear; the organisation still owns the decision to deploy, monitor, and review the system under its own control framework.

Where the system includes agentic capabilities, accountability should also extend to the identity and authority granted to the agent itself, including what it can access, change, or execute. That intersection between AI governance and non-human identity is increasingly important, but current guidance suggests it should be handled through explicit control ownership rather than assumed as part of application development. For regulated or safety-sensitive models, the approver should be named, recorded, and able to explain why the residual risk was acceptable at the time of release.

Related resources from NHI Mgmt Group

• What do regulators expect from AI and machine learning risk models?
• How should security teams govern machine learning models that may contain hidden backdoors?
• How should teams prevent bad data from reaching machine learning models?
• How should teams evaluate machine learning models when classes are highly imbalanced?