MIT’s AI risk study exposes the developer accountability gap

At a glance

What this is: MIT’s expert panel assigns primary AI security responsibility to developers, exposing a mismatch between formal accountability and how security actually fails in practice.

Why it matters: IAM, NHI, and AI governance teams need to understand that responsibility assignments do not create control, evidence, or enforcement when autonomous systems and delegated access are involved.

By the numbers:

• 88% of enterprises will be deploying agents by the end of 2026.
• 38% of businesses already have unauthorized agent deployments their security teams haven’t sanctioned.
• 48% of CISOs now name agentic AI as the top attack vector for 2026.

👉 Read ZioSec's analysis of MIT's AI risk responsibility study

Context

MIT’s AI Risk Initiative study is about responsibility allocation for AI security risk, but the practical issue for identity teams is simpler: who actually owns control when software can act, chain tools, and change behavior at runtime. In agentic environments, responsibility assigned on paper often fails to match the control plane that exists in production.

For IAM, NHI, and governance programmes, the gap matters because agentic systems do not fit neatly into static ownership models. If access can be delegated, combined, or exercised without tight oversight, then security can no longer depend on a job title, a sprint team, or a policy statement to keep the system safe.

Key questions

Q: What breaks when AI security responsibility is assigned only to developers?

A: Security breaks when responsibility is separated from runtime control. Developers may define an agent’s initial behaviour, but they do not continuously observe tool use, scope drift, or unsanctioned deployments. Without operational evidence and ownership, the programme becomes a paperwork model rather than a control model.

Q: Why do agentic AI systems complicate identity governance?

A: Agentic systems complicate identity governance because their access is not static. They can select tools, chain actions, and change behaviour while running, which means least privilege, recertification, and offboarding have to reflect runtime evidence rather than only design-time intent.

Q: How do security teams know whether AI agents are actually governed?

A: They know only if discovery, ownership, and telemetry are in place. A governed agent is one you can inventory, attribute, monitor, and retire. If any of those steps is missing, the organisation is managing assumptions rather than identities.

Q: Who should be accountable for AI agent risk in practice?

A: Accountability should sit with the team that can prove control over the agent’s lifecycle, access, and behaviour. That usually means shared operational responsibility between engineering, security, and governance, backed by telemetry and formal ownership, not a single named role in isolation.

Technical breakdown

Why developer ownership breaks down in agentic systems

The study’s premise assumes developers can meaningfully carry AI security responsibility because they define code, tool access, and failure modes. That is true only at design time. Once an agent is running, its actions depend on inputs, tool routing, and runtime decisions that are not fully visible to the original builder. This is why accountability and control drift apart in agentic environments. The developer may define the boundary, but the operational blast radius is shaped by execution. Practical security cannot rely on upstream intent alone.

Practical implication: treat runtime agent behaviour as the control point, not the developer’s original design intent.

Why unauthorized agent deployments create identity blind spots

The article’s unauthorized deployment figure points to a familiar identity problem: systems that can touch data, tools, or cloud services often exist outside formal inventory and approval flows. In identity terms, this is shadow AI with delegated access. If an agent is not discovered, it cannot be governed, recertified, or removed from service. That makes discovery and inventory the first security control, not an administrative clean-up task. Identity governance fails when the estate is incomplete.

Practical implication: build discovery and ownership mapping for every agent before you try to assign policy or accountability.

Why evidence matters more than responsibility statements

The article argues that responsibility must be backed by evidence because roles alone do not reduce risk. That is the right framing. In high-velocity AI environments, governance needs attestable proof of what an agent can access, what it did, and when its scope changed. Without that evidence, responsibility becomes ceremonial and incident response becomes retrospective guesswork. This is the same structural weakness seen in machine identity programmes that lack lifecycle visibility and usage telemetry.

Practical implication: require continuous evidence of access, actions, and scope changes for every agent identity.

Threat narrative

Attacker objective: The attacker aims to exploit unmanaged agent behaviour and delegated access to reach sensitive systems, data, or actions without effective governance.

1. Entry occurs when an unauthorised or weakly governed AI agent is deployed into production with access to tools, data, or APIs.
2. Escalation happens when the agent chains actions and expands its effective scope beyond what the original developer or deployer expected.
3. Impact follows when agentic behaviour reaches sensitive systems or data paths without timely oversight, creating a breach window that static responsibility models do not catch.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Developer responsibility is not the same thing as security control. The MIT panel is correct that developers shape AI risk at design time, but the security failure appears later, when the system operates beyond the developer’s direct oversight. That means ownership statements can be directionally right and operationally useless at the same time. The practitioner implication is clear: governance must target runtime evidence, not org-chart accountability.

Runtime governance gap: AI security fails when responsibility is assigned before execution but enforcement is absent during execution. This is the named concept the study points to. Agentic systems can select actions, call tools, and alter scope in ways the builder cannot fully predict, so the control problem shifts from authoring to observing. Practitioners should treat this as a structural gap in identity governance, not a documentation problem.

Unauthorized agent sprawl is the precursor to identity failure. If a security team cannot enumerate which agents exist, it cannot recertify, monitor, or retire them. The article’s discussion of unsanctioned deployments shows that the first breach condition is often invisibility, not overt abuse. The implication is that discovery and ownership mapping must precede any serious AI governance programme.

Accountability without telemetry becomes scapegoating. The article’s strongest point is that someone must own the risk, but ownership alone does not produce control evidence. When an agent changes behaviour, security teams need traces, attestations, and scope records that survive handoffs between development, deployment, and operations. Practitioners should judge AI governance by whether it produces auditable proof, not by whether it assigns blame cleanly.

Agentic AI collapses the old separation between builders and operators. Traditional software lets security teams review code, monitor runtime, and intervene with some stability in the middle. Agentic systems compress that window, which means the role that writes the system is no longer sufficient to secure the system. The field should therefore stop treating secure development as a substitute for operational identity governance.

From our research:

• 38% of businesses already have unauthorized agent deployments their security teams haven’t sanctioned, according to LLMjacking: How Attackers Hijack AI Using Compromised NHIs.
• DeepSeek accidentally embedded over 11,000 secrets in its training data and left a database exposed online, revealing more than one million sensitive records including chat histories, backend credentials, and API keys.
• That is why the OWASP Agentic AI Top 10 remains the right next reference point for teams building runtime controls around agent behaviour.

What this signals

Runtime governance gap: if your AI programme still treats developer responsibility as a substitute for operational control, you will miss the point where agent behaviour actually becomes risk. The governance test is whether you can discover, attribute, and attest every agent before it can act outside the intended boundary.

The scale signal is already visible. With 38% of businesses reporting unsanctioned agent deployments in the LLMjacking analysis, identity teams should assume shadow AI will be a standing inventory problem, not an edge case.

For practitioners, the next step is to align AI oversight with established identity controls, then extend them to runtime evidence. The OWASP Agentic AI Top 10 is a useful reference for that shift.

For practitioners

• Define runtime ownership for every agent Assign a named operational owner for each agent identity, including any agent that can call tools, access data, or act without direct human approval. The owner must be accountable for inventory, access scope, and retirement decisions.
• Discover unsanctioned agents continuously Run discovery across cloud, application, and workflow layers to find agents that were created outside formal approval flows. Compare what exists with what is registered, then close the gap before policy enforcement starts. Use the ultimate guide to NHIs for lifecycle framing and the OWASP NHI Top 10 for control priorities.
• Instrument agents with attestable evidence Capture logs that show what each agent accessed, what tool it invoked, and when scope changed during execution. Evidence should be reviewable by security, audit, and governance teams, not just by engineers.
• Rebuild recertification for autonomous behaviour Do not recertify agent access on the same cadence as human roles if the access can change during a session. Rework reviews around actual behaviour, observed scope, and service ownership rather than static entitlements.

Key takeaways

• Developer accountability is a useful label, but it is not a security control unless the organisation can enforce access and prove runtime behaviour.
• The strongest warning sign in the article is the gap between formal responsibility and unsanctioned deployment, which shows that AI governance is already out of sync with reality.
• Practitioners should move from assigning blame to collecting evidence, because identity governance only works when agents can be discovered, attributed, and retired.

Key terms

• Agentic AI: Software that can choose actions, call tools, and alter execution based on runtime conditions. In identity governance, agentic AI matters because access is no longer just granted and used. It is continuously selected, combined, and exercised by the system itself, which creates a different control problem from static automation.
• Runtime governance gap: The gap between who is assigned responsibility for a system and who can actually observe and control its behaviour while it is running. For AI systems, this gap grows when access decisions, tool selection, and action timing happen after deployment, outside the builder’s direct oversight.
• Shadow AI: AI systems or agents operating without formal approval, inventory, or governance. In practice, shadow AI is not just an asset management issue. It is an identity problem because unmanaged agents can still hold credentials, access tools, and create security exposure without a known owner.
• Runtime evidence: Auditable proof of what an identity did, what it accessed, and when its scope changed during execution. For agentic and non-human identity programmes, runtime evidence is what turns policy from a statement into a control because it allows security teams to verify behaviour after deployment.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.

This post draws on content published by ZioSec: MIT asked 272 experts who owns AI risk, and they picked developers. Read the original.