How do organisations decide who is accountable for AI security failures?

Why This Matters for Security Teams

AI security failures are rarely caused by a single broken control. They usually emerge when ownership is fragmented across the model, the data pipeline, the workflow, and the enforcement layer. That makes post-incident questions harder than they should be: who approved the change, who monitored the behaviour, and who can actually remediate it?

Current guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls and the CSA MAESTRO agentic AI threat modeling framework both point to the same operational truth: security accountability has to map to a specific control surface, not to a vague AI initiative. NHIMG research shows why this matters in practice. In the State of Non-Human Identity Security, only 1.5 out of 10 organisations said they were highly confident in securing NHIs, which is a warning sign for any AI stack that depends on machine credentials, API keys, and delegated access.

In practice, many security teams encounter accountability gaps only after an AI workflow has already leaked secrets, overreached permissions, or produced an uncontainable incident, rather than through intentional control design.

How It Works in Practice

Accountability works best when it is assigned by control plane, not by project label. The model owner is accountable for training inputs, model behaviour, and release decisions. The dataset owner is accountable for provenance, retention, and contamination risk. The workflow owner is accountable for how the system is used in production, including prompts, tool calls, and escalation paths. The enforcement owner is accountable for policy, identity, and runtime guardrails.

This structure is especially important when AI systems operate with non-human identities, because secret sprawl and delegated access create fast-moving blast radius. NHIMG research on LLMjacking and the DeepSeek breach shows how exposed credentials and embedded secrets can turn a model or toolchain into an access path for attackers. That is why ownership should include:

• named control owners for each AI component
• runtime logging and audit trails that show who approved and who changed what
• separation between development responsibility and production enforcement
• incident playbooks that identify whether the failure was in data, model logic, or access control

For agentic systems, accountability also depends on real-time policy evaluation. A policy in the Anthropic Project Glasswing style, or in a policy engine aligned to NIST controls, should decide whether a tool call is allowed at the moment of request. That is more effective than assigning broad responsibility after the fact, because autonomous systems can chain tools and change state faster than static approvals can keep up. These controls tend to break down in highly distributed environments where model hosting, retrieval, and action execution are owned by different teams with inconsistent logging standards.

Common Variations and Edge Cases

Tighter accountability often increases coordination overhead, requiring organisations to balance clearer ownership against slower delivery and more governance work. That tradeoff is real, especially in shared platforms where one team operates the model and another team owns the business workflow.

Best practice is evolving for edge cases. In vendor-hosted AI services, accountability often splits between the customer and the provider, but there is no universal standard for this yet. The customer usually remains accountable for data, access, and safe use, while the provider may own platform controls and service resilience. In regulated environments, teams should document this split explicitly in contracts, architecture diagrams, and control matrices.

Another common edge case is multi-agent orchestration. One agent may call tools, another may retrieve data, and a third may approve actions. That makes a generic AI owner insufficient. The more useful question is which component can create, approve, or execute a harmful action. NIST AI risk guidance and CSA MAESTRO both support this component-level view, but the exact boundary assignment is still a governance decision, not a fixed industry standard. Organisations that skip that decision usually discover the accountability gap after an incident has already crossed system boundaries.

Related resources from NHI Mgmt Group

• What is MCP in the context of AI security?
• What are the security risks associated with AI agents?
• How should security teams govern machine identity credentials in agentic AI environments?
• How should security teams manage permissions for AI agents?