AI agent autonomy and agency are widening security risk faster than controls

At a glance

What this is: This is an analysis of how AI autonomy and agency expand the attack surface when LLMs gain memory, tool access, and self-directed execution.

Why it matters: It matters because IAM, PAM, and AI security teams need to constrain what AI systems can access and do before agent behaviour outpaces governance.

By the numbers:

• Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027.
• AI spending is to hit USD 375 billion this year alone, and USD 500 billion next year.
• A cybercriminal targeted 17 organizations as a part of a data extortion operation that sometimes exceeded USD 500,000 in the demanded ransoms.
• Only 44% have implemented any policies to govern AI agents, despite 92% agreeing governance is critical to enterprise security.

👉 Read Promptfoo's analysis of autonomy, agency, and AI security risk

Context

AI agent autonomy becomes a governance problem when systems can choose actions, use tools, and retain memory outside tightly defined human approval paths. The article frames that shift clearly: the issue is not whether LLMs are human-like, but whether they can operate with enough independence to outpace existing access controls and accountability models.

That matters for identity programmes because AI systems increasingly behave like non-human identities with permissions, sessions, and delegated access that must be bounded. Once an agent can browse, call APIs, write files, or trigger workflows, the security question becomes who authorises the action, who owns the identity, and how the blast radius is constrained when the model misbehaves.

Key questions

Q: How should security teams govern AI agents that can choose tools at runtime?

A: Security teams should govern runtime agent choice as an access event, not as a simple application action. That means scoping permissions to the task, limiting token lifetime, logging every tool decision, and blocking the agent from reaching systems outside its approved context. Static roles alone are not enough when the execution path changes on each run.

Q: Why do AI agents create more risk than traditional automation?

A: AI agents create more risk because they can interpret context, choose actions, and invoke tools autonomously. Traditional automation follows fixed rules, but an agent can be manipulated into using its own authority in unintended ways. That makes permission scope, tool boundaries, and monitoring more important than model accuracy alone.

Q: What breaks when AI agents are given broad standing access?

A: Broad standing access breaks governance because the agent can move from one task to another without a fresh authorization check. That creates a control gap between intended scope and actual runtime behaviour. The result is weak accountability, limited containment, and audit trails that show activity without explaining why the activity was allowed.

Q: Who is accountable when an AI agent takes a harmful action in healthcare?

A: Accountability should remain with the human or team that deployed and authorised the agent, not with the model itself. The organisation needs named ownership, scope definitions, and logs that tie each action to an identity. Without that chain of responsibility, agentic behaviour becomes operationally opaque and difficult to defend in audits or investigations.

Technical breakdown

How memory and tool use turn LLMs into higher-risk systems

LLMs become materially riskier when memory, tool calls, and chained prompts combine. Memory lets a system persist context across sessions, tool use gives it the ability to act through APIs, and chained reasoning creates longer execution paths that are harder to inspect. That is not autonomy in a human sense, but it is enough to create software that can pursue goals, repeat actions, and amplify mistakes. The real risk is not sentience. It is that the system can act at machine speed with access that was never intended for open-ended use.

Practical implication: Treat every tool-enabled model as a governed workload with explicit permissions, not as a conversational interface with implied trust.

Why unbounded agency expands the attack surface

Agency matters because it turns model output into action. When an LLM can browse, execute code, write to storage, or call internal services, prompt injection and malicious instructions can move from misleading text to operational damage. The broader the tool set, the more places an attacker can steer the system. This is why the article’s focus on confinement, input filtering, and output validation is technically sound. Security controls must assume the model will sometimes generate bad decisions and that the surrounding environment must prevent those decisions from becoming incidents.

Practical implication: Limit tool scope, isolate execution, and require step-up approval before any model action reaches a sensitive system.

Autonomy drift and the problem of accountability

Autonomy drift occurs when a system gradually behaves outside the decision boundary originally intended for it. In AI environments this can happen through recursive prompting, overbroad tool permissions, or workflows that let the model keep acting after the original task changes. Unlike human staff, these systems do not carry legal or moral responsibility, so accountability must sit with the deployer and the control plane around the model. That makes governance, auditability, and kill-switch design foundational security requirements, not optional safeguards.

Practical implication: Define ownership, logging, and emergency shutdown paths before enabling long-lived or self-invoked agent workflows.

Threat narrative

Attacker objective: The attacker wants to turn AI assistance into a controllable execution layer for fraud, data theft, or harmful automation.

1. Entry begins when a malicious prompt, unsafe integration, or overbroad API connection reaches an AI system with tool access.
2. Escalation occurs when the model uses memory, recursive prompting, or delegated credentials to expand beyond its intended task boundary.
3. Impact follows when the system exfiltrates data, executes unsafe code, or enables fraud, ransomware assistance, or other downstream abuse.

NHI Mgmt Group analysis

Autonomy without bounded identity is a governance failure, not a product feature. Once an AI system can select actions and invoke tools, it needs the same access discipline applied to other privileged software. The article is right to push back on anthropomorphic framing because the operational question is identity, permissions, and control scope. For practitioners, the implication is clear: if you cannot bound the agent, you cannot safely deploy it.

Named concept: agentic control leakage. This is the gap between what an AI system is allowed to decide and what it is allowed to do. Tool access, memory, and recursive prompting can leak control beyond the original task, even when the model appears to be functioning normally. That makes control-plane design central to AI governance. Practitioners should treat every delegated action as a privilege decision, not a model output.

AI systems now sit inside the same governance problem space as non-human identities. The article’s warning about human-level caution maps directly to NHI governance because agents increasingly depend on credentials, tokens, and service access. The lesson is not that agents are humans. It is that their access patterns create the same lifecycle questions that service accounts and workloads already pose. Teams should align AI security with identity lifecycle controls rather than bolting it onto model policy alone.

The market will keep rewarding capability, but security programmes will be judged on restraint. The article notes that value must outpace risk and cost, and that is exactly where governance will matter most. Organisations that can explain why a model has a given permission set, what stops it from exceeding scope, and who can cut it off will be better placed than those relying on informal trust. Practitioners should expect AI adoption to slow where access governance is weak.

Gartner’s forecast of over 40% project cancellation is a signal of control debt, not just hype fatigue. That cancellation risk reflects the friction between ambitious AI use cases and unresolved safety, access, and accountability gaps. Projects fail when governance arrives after deployment instead of before it. For practitioners, the conclusion is that access control architecture must be part of the business case, not a retrofit after the pilot stage.

What this signals

Agentic control leakage will become a practical governance category for security teams, because the failure mode is no longer model accuracy alone but permission expansion through tools, memory, and delegated execution. Teams that already manage service accounts and privileged workflows have the right mental model, but they need to apply it to model-driven systems now.

The operational signal to watch is not whether AI is being adopted, but whether every agent can be tied to a narrow identity, an auditable action trail, and a revocation path. That is where identity governance and AI governance overlap most directly, and it is where many programmes will discover control debt first.

The article points toward a future in which trust in AI systems will be judged less by how fluent they sound and more by how tightly their access is governed. For practitioners, that means pairing agent rollout with access review, step-up controls, and policy enforcement at the execution layer, not just in the interface.

For practitioners

• Define agent permission boundaries before deployment Map every model and agent to a specific identity, then grant only the APIs, files, and workflows required for the task. Remove ambient access and document the approval path for any privilege expansion.
• Constrain tool use with step-up approval Require human confirmation before a model can call sensitive systems, move data, or trigger irreversible actions. Use policy checks at the execution layer, not only at the prompt layer.
• Log and audit every agent action trail Capture prompts, tool calls, outputs, and delegated credentials in a tamper-evident audit trail so investigations can reconstruct what the system actually did.
• Add kill-switches and time-bound sessions Design a rapid shutdown path for self-invoked processes and expire access when the task closes. Do not allow long-lived autonomous sessions to persist beyond the original business need.
• Test for autonomy drift and prompt injection Red team workflows that combine memory, recursion, and external tools so you can see where the agent crosses its intended decision boundary under malicious input.

Key takeaways

• AI agents become a security problem when memory, tool use, and delegated action combine into persistent control.
• The evidence points to a real governance gap, with broad recognition of risk but far fewer organisations enforcing policies or auditability.
• Security teams should treat AI systems as governed identities with constrained permissions, short-lived access, and revocation paths.

Key terms

• Agentic AI: Autonomous AI systems capable of planning, deciding, and taking actions — including calling APIs, writing code, and orchestrating other agents — with minimal human oversight. Agentic AI introduces new NHI risks as agents must authenticate to external services.
• Autonomy Drift: The gradual movement of an AI system outside the decision boundary it was originally given. It can happen through recursive prompting, memory, or overly broad tooling, and it becomes a security issue when the system keeps acting beyond the task that justified its access.
• Tool Use Module: A tool use module is the part of an agentic system that connects the agent to external systems such as APIs, search engines, code execution, or browser automation. It is where identity risk becomes operational because the agent can cross trust boundaries and trigger side effects beyond the prompt itself.
• Agentic Control Leakage: The point at which an AI system’s allowed decision space expands into actions it was not meant to perform. It is the security gap between intention and execution, and it often appears when permissions, memory, and execution paths are not tightly bounded.

What's in the full article

Promptfoo's full article covers the operational detail this post intentionally leaves for the source:

• The article expands on practical limits for memory, tool access, and recursive prompting that determine whether an AI system can act safely.
• It outlines concrete containment measures such as sandboxing, input filtering, output validation, and human confirmation before access is granted.
• It discusses real-world misuse patterns, including ransomware support, fraud, fake identity creation, and extortion workflows.
• It frames the accountability problem for autonomous behaviour and why designers, deployers, and users all need explicit responsibility boundaries.

👉 The full Promptfoo article expands on autonomy limits, agent misuse, and containment controls for AI systems.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security practitioners build the access and lifecycle controls needed for AI systems, workloads, and other non-human identities.