Who is accountable when an AI security control fails because a model is switched off?

Why This Matters for Security Teams

When an AI security control depends on a model that can be switched off, the real issue is not only technical availability. It is accountability for the decision to rely on that control in production, especially when a failure can weaken detection, blocking, or approval workflows. Security teams should treat model dependency as a control design issue, not just a service uptime issue, and align it with third-party risk, change management, and resilience planning. NIST’s NIST SP 800-53 Rev 5 Security and Privacy Controls is useful here because it frames controls around governance, contingency, and supplier oversight rather than assuming a tool will always remain available.

The accountability question becomes sharper when the model is not owned by the buyer, or when the model provider changes behaviour, deprecates an endpoint, or removes a capability without a clear notice period. In those cases, the security team still has to answer whether the organisation accepted that dependency, documented the fallback, and tested the failure path. This is especially important for AI-supported policy checks, content inspection, and agent guardrails, where a silent outage can create a false sense of protection. In practice, many security teams discover this only after an alerting gap or control outage has already affected operations, rather than through intentional resilience testing.

How It Works in Practice

In practice, accountability should be divided across three layers: the vendor, the buyer, and the internal control owner. The vendor is accountable for service delivery, notice of material changes, and published support boundaries. The buyer is accountable for deciding whether the control is critical, whether the fallback is acceptable, and whether the risk is formally accepted. The internal owner, often security architecture or the platform team, is accountable for ensuring the control degrades safely.

A practical implementation usually includes contract language, technical monitoring, and operational runbooks. Teams should define what happens if the model returns errors, timeouts, altered outputs, or complete unavailability. They should also document whether the control fails open, fails closed, or reroutes to a human review path. For agentic systems, this matters even more because a disabled model may affect planning, tool use, or approval logic. The CSA MAESTRO agentic AI threat modeling framework is useful for mapping those dependency failures into the broader system risk picture.

• Classify the model-backed control by business criticality and recovery expectation.
• Define a fallback path for unavailability, degradation, and provider modification.
• Log model status separately from security outcomes so failures are visible in operations.
• Include service removal, version change, and policy drift in third-party risk reviews.
• Test the control during exercises, not only during procurement or annual review.

Where organisations need stronger operational proof of resilience, it is sensible to compare this with emerging provider practices such as Anthropic’s Project Glasswing, which reflects the growing emphasis on explicit model lifecycle control. These controls tend to break down when the AI model is embedded inside a vendor-managed workflow platform because the buyer cannot independently observe failure, capture alerts, or force a safe fallback.

Common Variations and Edge Cases

Tighter model dependency controls often increase operational overhead, requiring organisations to balance resilience against deployment speed and vendor convenience. There is no universal standard for this yet, so current guidance suggests treating model availability as part of control assurance rather than assuming it is just an infrastructure concern.

One common variation is the difference between a model used for advisory output and a model used to enforce a security decision. If the model only assists a human, the fallback may be a manual review queue. If the model blocks access, approves code, or suppresses alerts, the fallback must be more rigorous and usually needs a tested alternate control. Another edge case appears when a vendor changes the model silently but keeps the endpoint live. In that situation, the control may still be “up” while its behaviour has drifted, which creates a different accountability problem.

Model switching is also more complex in regulated environments where auditability matters. Security leaders should be explicit about who signs off on accepted degradation, how long a fallback can remain in place, and whether any compensating control has comparable assurance. In agentic deployments, that often means naming a human control owner for the AI behaviour itself, not just the hosting service. The key point is that availability, integrity, and governance are linked, and none of them should be treated as someone else’s problem once the model goes live.

Related resources from NHI Mgmt Group

• How should security teams balance agility with identity control in cloud and AI environments?
• How should security teams govern AI agents that use Model Context Protocol?
• How should security teams govern AI agents using Model Context Protocol?
• What is the difference between model guardrails and runtime AI security controls?