MIT’s AI risk study exposes the developer accountability gap

At a glance

What this is: MIT's AI Risk Initiative study says developers are viewed as primarily responsible for AI security risk, but the article argues that accountability and operational control do not line up.

Why it matters: IAM, PAM, and AI governance teams need to account for the gap between formal responsibility and the role that can actually enforce controls across AI and agentic systems.

By the numbers:

• MIT AI Risk Initiative surveyed 272 experts on who should bear responsibility for AI risk.

Context

AI security risk is a governance problem when the role assigned responsibility is not the role with the power to enforce controls. The article argues that developer accountability becomes unstable once agentic systems can change behaviour, chain tools, and act outside the assumptions of traditional software delivery.

MIT AI Risk Initiative’s Delphi study is used as the trigger for that argument, not the conclusion itself. The central issue for identity and access programmes is that AI systems create a responsibility model on paper and an execution model in production, and those two models can drift apart quickly.

For identity teams, the important question is not just who is blamed after failure. It is which actor actually controls authorisation boundaries, observability, and evidence when the system makes runtime decisions that no one reviewed in advance.

Key questions

Q: What breaks when AI security responsibility is assigned only to developers?

A: Security breaks when responsibility is separated from runtime control. Developers may define an agent’s initial behaviour, but they do not continuously observe tool use, scope drift, or unsanctioned deployments. Without operational evidence and ownership, the programme becomes a paperwork model rather than a control model.

Q: Why do AI users score low as owners of AI security risk?

A: AI users can initiate use, but they usually cannot govern tool choice, delegation, memory, or execution paths. That makes them poor proxies for security ownership when the real risk comes from runtime decisions. Security teams should separate interaction from control and assign responsibility to the actor that can enforce boundaries.

Q: How can organisations tell whether AI governance is actually working?

A: Organisations can tell AI governance is working when they can inventory every agent, explain its purpose, show who owns it, and prove that permissions are tightly scoped. If those four things are missing, the programme has policy language but not operational control. Auditors will notice the gap quickly.

Q: What is the difference between responsibility on paper and control in practice for agentic AI?

A: Responsibility on paper is a governance assignment. Control in practice is the ability to limit tools, set boundaries, observe actions, and revoke authority when needed. For agentic AI, those are not the same thing, and programmes that confuse them will misplace accountability while leaving the real risk untouched.

Technical breakdown

Why AI security responsibility breaks when the actor can change at runtime

In traditional software, developers can be treated as the party closest to the control surface because behaviour is comparatively stable after release. Agentic systems are different because they can choose tools, alter execution paths, and vary outcomes based on live inputs. That means responsibility assigned at design time may not match the actor that actually exercised authority at runtime. Once decision-making is no longer fixed, the control problem shifts from static ownership to continuous governance over what the system can do, when it can do it, and under what evidence.

Practical implication: Treat runtime authority as the real control boundary, not the team name on the chart.

Why AI users are not the right proxy for AI security control

The study’s low responsibility score for AI users matters because usage is not the same as control. A user can trigger a system, but still have no visibility into prompt chaining, tool selection, memory effects, or downstream agent actions. That creates a common governance error: assigning accountability to the person who interacts with the system instead of the operator who can constrain identity, privilege, and telemetry. In AI and NHI programmes, proximity to the interface is not the same as ability to govern the risk.

Practical implication: Separate system use from control ownership when assigning governance duties.

Agentic AI identity requires evidence, not trust in developer intent

The article’s strongest operational point is that intent does not survive contact with production. For agentic systems, the security question is whether the runtime identity can be observed, constrained, and audited as it acts, not whether developers intended it to behave safely. That is the same structural problem NHI teams face with service accounts and APIs, but with faster and less predictable execution. A trust model based on prior approval or developer diligence is too weak once the system can initiate chained actions on its own.

Practical implication: Shift governance from stated intent to attestable runtime behaviour.

Threat narrative

Attacker objective: The objective is to turn runtime autonomy and weak governance boundaries into unauthorized access, unsafe actions, or breach conditions.

1. Entry begins when an agentic system is granted operational access and can initiate work through developer-defined tools and credentials.
2. Escalation occurs when the system chains tools or extends its behaviour beyond the original design assumptions during live execution.
3. Impact follows when this runtime drift creates unauthorized actions, unseen exposure, or security failures that no human reviewed in time.

Breaches seen in the wild

• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.
• Langflow Flodrix botnet 2025: Attackers used Langflow's unauthenticated code endpoint to dump AI servers' environment variables and enlist them in the Flodrix botnet.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Developer accountability is not the same as developer control. MIT AI Risk Initiative’s expert consensus shows where responsibility is expected to sit, but not where enforcement actually happens. In mature identity programmes, accountability only matters when the assigned actor can continuously constrain access, observe behaviour, and prove control. The practitioner conclusion is that AI governance must be built around enforceable runtime authority, not role labels.

There is an accountability vacuum between the developer, the deployer, and the CISO. The article is right to point out that security failures rarely land where the technical decision was made. That mismatch already exists in human IAM and NHI programmes, where ownership, operation, and blame often sit in different boxes. The implication is that agentic AI will amplify an old governance flaw unless runtime evidence closes the gap.

Assumption collapse: responsibility follows proximity to the code. That assumption was designed for software whose behaviour is comparatively stable after release. It fails when the actor is autonomous because tool choice, execution timing, and action sequence are decided in runtime rather than frozen at deployment. The implication is that security programmes must stop treating development proximity as a reliable proxy for control.

Agentic AI turns responsibility into an evidence problem. The article’s strongest contribution is showing that consensus on ownership does not solve governance when systems can act beyond human review cycles. That is the same structural lesson NHI teams learned with service accounts and tokens: if the actor can move faster than review, ownership alone is not control. Practitioners need evidence that matches machine-paced execution.

Identity blast radius: when an AI system can touch tools, data, and downstream agents, the security question becomes how far its authority can spread before anyone notices. That is a governance boundary problem, not a policy-wording problem. The practitioner conclusion is to measure the blast radius of runtime access, then govern that boundary as seriously as any privileged account.

From our research library:

• Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.
• Read next: Agentic AI Identity Maturity Model

What this signals

Developer proximity is not a reliable security control when the system can decide and act at runtime. That is the central programme lesson for agentic AI governance. If your control model assumes the person who built the system can also govern its live behaviour, your identity programme is already over-trusting intent and under-weighting enforcement.

Agentic systems force IAM and NHI teams to think about evidence at the speed of execution. Runtime authority, not design documentation, becomes the thing auditors and incident responders need to see.

Identity blast radius will become the more useful planning concept than simple ownership. Once an AI system can chain tools or affect downstream agents, the question is not only who is responsible, but how far its authority can reach before control is regained.

For practitioners

• Map runtime authority, not job titles Identify which team can actually constrain agent tools, credentials, and execution paths at runtime, then align ownership to that control surface.
• Inventory every deployed agent and agent-like workflow Create a complete list of AI systems that can call tools, move data, or trigger actions without a human approval gate, including unsanctioned deployments.
• Require continuous behavioural evidence Use live testing, logging, and attestation to prove what an agent did in production, rather than relying on developer intent or annual review cycles.
• Align governance with runtime privilege boundaries Set policy on the authority an agent can actually exercise, including tool access, data reach, and delegation scope, so accountability matches execution.

Key takeaways

• The article shows a governance mismatch: developers are assigned AI security responsibility, but that does not mean they control runtime behaviour.
• The risk grows as agentic systems scale because their actions can move faster than traditional review cycles and accountability structures.
• Security teams need runtime evidence, authority boundaries, and observable control points, not just role-based responsibility statements.

Key terms

• Runtime authority: Runtime authority is the permission an AI system has while it is actively deciding and acting, not just when it is approved. In governance terms, it is the point where access, tool use, and action scope become operational, which is why build-time review alone cannot prove safety.
• Accountability Gap: The accountability gap is the distance between the authority a machine identity can exercise and an organisation's ability to identify, govern, observe, and stop that authority. It appears when ownership, scope, and runtime evidence are not connected well enough to answer who acted and why.
• Identity Blast Radius: The amount of damage a compromised identity can cause across systems, data, and infrastructure. In NHI environments, it is shaped by permissions, network reach, and administrative capability rather than by the credential alone. Reducing blast radius is a containment strategy that limits lateral movement and data exposure.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.