Who is accountable when an AI system makes a harmful decision?

Why This Matters for Security Teams

Accountability is not a philosophical debate once an AI system can take action, call tools, or trigger downstream workflows. The practical question is who had authority over the identity that executed the decision, because that is where audit evidence, access review, and incident response need to land. Current guidance from the NIST Cybersecurity Framework 2.0 and NIST AI governance work points to clear ownership, but many organisations still stop at model ownership and miss the operational chain behind the action.

That gap matters because autonomous systems do not behave like human operators. They can chain prompts, reuse tokens, and invoke tools in ways that are hard to predict after deployment. When the organisation cannot trace who approved the permissions, who provisioned the secret, and which agent or service account executed the harmful step, accountability becomes diluted across teams instead of being actionable. NHIMG research on the DeepSeek breach shows how exposed credentials and leaked records quickly turn identity weakness into security failure, which is exactly why identity chain clarity matters for AI systems too.

In practice, many security teams encounter the accountability problem only after an AI-triggered action has already caused data exposure, privilege misuse, or a regulatory reportable incident.

How It Works in Practice

Accountability should be mapped to the identity chain that enabled the action, not just to the model vendor or the business unit that bought the software. For autonomous or agentic systems, that chain usually includes the human owner, the platform team, the policy owner, the delegated agent, and any tool or workload identity used at execution time. The emerging best practice is to treat the agent as an execution subject with bounded authority, then record every runtime decision that expands, narrows, or consumes that authority.

That means pairing least privilege with workload identity, short-lived credentials, and policy evaluation at request time. The NIST Cybersecurity Framework 2.0 is useful here because it reinforces governance, access control, and continuous risk management, while the DeepSeek breach is a reminder that exposed secrets and uncontrolled data paths become a direct accountability problem, not just a confidentiality issue. In agentic environments, static RBAC is often too blunt on its own; current guidance suggests combining RBAC with intent-based authorisation so the system checks what the agent is trying to do, with what data, and in what context.

• Assign a named business owner for the agent’s permitted outcomes.
• Bind the agent to a workload identity, not a shared human credential.
• Issue JIT credentials and revoke them when the task ends.
• Log the policy decision, tool call, and approval path for each high-risk action.
• Keep secrets ephemeral so post-incident review can reconstruct use without exposing long-lived access.

When this is done well, accountability becomes a traceable control function rather than an after-the-fact blame exercise. These controls tend to break down when teams allow shared service accounts, long-lived API keys, or unsupervised tool chaining because the execution path no longer matches the approved identity chain.

Common Variations and Edge Cases

Tighter identity controls often increase operational overhead, so organisations must balance traceability against developer friction and runtime latency. That tradeoff is especially visible when multiple agents collaborate or when one agent delegates tasks to another, because each hop needs its own accountable identity and policy record. There is no universal standard for this yet, but the direction of travel is clear in NIST Cybersecurity Framework 2.0, DeepSeek breach lessons, and current agentic ai security work.

Edge cases matter. If an AI system is only suggesting actions and a human clicks “approve,” accountability sits primarily with the human approver and the control owner for the workflow. If the system can execute without human review, the accountability burden shifts toward the platform team that defined the guardrails and the business owner that accepted the risk. If the system uses third-party tools, identity and liability can fragment across vendors unless contracts and logs preserve the execution chain. Best practice is evolving here, so organisations should document the decision rule in advance: who approves, who monitors, who can revoke, and who is notified when the agent deviates from expected intent.

In practice, the hardest failures appear in environments with shared orchestration layers, copied secrets, and unclear approval boundaries, because the organisation can no longer prove which identity actually made the harmful decision.

Related resources from NHI Mgmt Group

• Who is accountable when an AI system in finance makes a policy-relevant decision?
• Who is accountable when an AI agent makes a risky decision?
• Who is accountable when a customer-facing AI gives harmful or off-topic advice?
• What makes agentic AI an NHI governance issue?