AI agent governance is an unsolved identity problem

At a glance

What this is: This roundtable argues that AI agents are already operational, but the governance model around identity, liability, and runtime control is still immature.

Why it matters: For IAM, NHI, and security teams, the issue is not whether agents exist but whether current access models can govern non-human actors that act, delegate, and fail differently from people.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, sharing sensitive data, or revealing credentials.
• Only 44% of companies have implemented policies to govern AI agents, even though 92% say that governing them is critical to enterprise security.

👉 Read FireCompass's discussion of AI agent governance and runtime control

Context

AI agent governance is the practice of controlling software entities that can make decisions, select tools, and execute actions inside an organisation. The identity problem is that these actors do not map cleanly to human IAM, because their access patterns, error modes, and escalation paths can change within a single session.

The roundtable reflects a wider industry transition: agents are being treated like productive workers before most enterprises have inventory, ownership, runtime enforcement, or review discipline in place. That is why the most useful discussion is no longer about whether agents are useful, but about which identity assumptions break first when they are allowed to do real work.

For practitioners, the key question is not how to give agents more access. It is how to govern per-agent identity, limit tool reach, and preserve auditability when actions are generated dynamically rather than pre-scripted.

Key questions

Q: What breaks when AI agents are treated like standard human users?

A: You lose visibility into effective permissions, expected behaviour, and real blast radius. Human-centric controls can misclassify normal agent activity as compromise, or miss policy violations that happen entirely within legitimate access. The failure is not only technical, it is governance design that assumes a person is always behind the action.

Q: Why do AI agents create a liability problem for organisations?

A: Courts are already leaning toward the deploying organisation being responsible for what its agents say or do. That shifts the problem from proving intent to proving control, ownership, and evidence. If the organisation cannot show who owned the agent, what it could access, and how actions were logged, liability becomes much harder to manage.

Q: What are the signs that AI governance is failing in the enterprise?

A: Common warning signs include rapid growth in AI use without matching policy coverage, sensitive files being copied into personal accounts, and a large share of AI apps carrying high or critical risk. Another indicator is weak visibility into who is using which tools and what data they are sending. If teams cannot answer those questions, governance is not working as intended.

Q: How should security teams govern AI agents that inherit authority from other identities?

A: Security teams should govern AI agents by tracking identity lineage, not just credentials. That means recording the originating identity, the delegated authority path, and the runtime context for each action. If an agent can inherit permissions from humans, services, or other agents, policy has to evaluate the full chain before access is granted or continued.

Technical breakdown

Per-agent identity and runtime access control

The roundtable converges on a basic architectural point: AI agents need identity at runtime, not just a label in a registry. Per-agent identity means each agent must be uniquely known, scoped, and logged so its actions can be tied back to a responsible owner and use case. Runtime access control matters because the agent’s actual tool use and data access are decided during execution, not fully at provisioning time. That makes static approval alone insufficient, especially where the agent can chain steps, call tools, and trigger side effects without a human in the loop.

Practical implication: treat each agent as a governed identity with explicit runtime boundaries, not as a generic automation job.

Why non-determinism breaks auditability in AI agents

A human reviewing an event can usually reconstruct intent and sequence. An AI agent can produce different outputs from the same prompt, even when asked the same question twice. That non-determinism is not just a quality issue, it creates an audit problem because the control record may not match the exact path the agent took. If the organisation cannot reproduce the decision path, it cannot reliably prove why an access request, recommendation, or action occurred. Deterministic logging and complete action traces become essential once the agent can influence real systems.

Practical implication: require deterministic logs and replayable traces before allowing agents to touch sensitive workflows.

Tool segregation and destructive action gating

The strongest control pattern described in the article is to separate planning from action. An agent may be able to reason about what should happen, but if it cannot reach the required tool, it cannot cause the outcome. That is why the article emphasises input and output firewalls, capability control, and deterministic enforcement for destructive actions. This is an identity control as much as a security control, because the question is which actions the actor is allowed to take, under what conditions, and with what approval boundaries.

Practical implication: gate irreversible actions behind explicit approval and remove disruptive tools from the agent’s reachable scope.

Threat narrative

Attacker objective: The objective is to turn trusted agentic access into repeatable, high-speed damage that is difficult to attribute, contain, or audit.

1. Entry begins when an AI agent is granted legitimate organisational access and can operate with enough privilege to interact with real tools and data sources.
2. Escalation occurs when the agent’s runtime behaviour drifts beyond the intended task, either by chaining actions, creating more agents, or reaching tools that were not tightly constrained.
3. Impact follows when repeated mistakes, tool misuse, or unaudited actions amplify at machine speed and produce organisation-wide liability, data exposure, or operational loss.

Breaches seen in the wild

• Meta AI Instagram Account Takeover — 20,225 Instagram accounts hijacked via compromised Meta AI support chatbot with overprivileged access.
• Replit AI Tool Database Deletion — Replit vibe coding AI assistant deletes live production database and creates 4,000 fake user records.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI agent governance is now an identity discipline, not an AI feature discussion. The roundtable makes clear that agents must be handled as privileged identities with ownership, scope, and audit boundaries. That places the problem squarely inside IAM, PAM, and NHI governance rather than leaving it in a general AI policy lane. Security teams that separate agent governance from identity governance are already misclassifying the risk.

Per-agent identity is the minimum viable control model for agentic systems. A shared pool of unnamed agents creates attribution gaps, weak accountability, and impossible review workflows. The article’s strongest point is that identity must map to execution, not just deployment. Practitioners should read this as a call to govern agents individually, because aggregate oversight cannot explain individual runtime behaviour.

Assumption collapse is the real story here: access review processes were designed for stable access that lasts long enough to be observed and recertified. That assumption fails when an autonomous system can acquire, combine, and discard privileges during active execution. The implication is not merely that controls need tuning, but that lifecycle governance built around human-paced review cycles does not match agent-timed behaviour.

Shadow AI is a governance multiplier, not just a discovery problem. The article’s warning that approved agents can multiply into many more agents means inventory alone is not enough. Once agents can create agents, oversight moves from counting assets to controlling delegation chains, which is a materially harder governance problem for NHI and IAM programmes. Practitioners should expect discovery to lag behaviour unless runtime controls are already in place.

Non-determinism changes what evidence means. If the same prompt can yield different answers, then traditional audit expectations built around stable, repeatable workflows become weaker. That has direct implications for compliance, incident review, and accountability, because proving what an agent did becomes as important as constraining what it can do. Security leaders should treat reproducibility as a governance requirement, not a nice-to-have.

From our research:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, sharing sensitive data, or revealing credentials, according to AI Agents: The New Attack Surface report.
• 52% of companies can track and audit the data their AI agents access, which means 48% still face a complete compliance and breach-investigation blind spot.
• For a broader identity control perspective, see OWASP Agentic AI Top 10 for the agent-tool and privilege risks practitioners need to map next.

What this signals

Identity governance for AI agents will increasingly be measured by runtime evidence, not by policy statements. Enterprises can write policies quickly, but the harder task is proving that every agent action is attributable, bounded, and reversible. The organisations that will move fastest are the ones that connect agent inventory to access reviews, approval gates, and deterministic logging before scale makes those controls harder to retrofit.

Per-agent identity and lifecycle discipline will become the deciding control layer. Once agents can create more agents, the governance unit is no longer the platform, it is the actor. That means IAM and NHI teams need one control story for provisioning, ownership, review, and retirement across agentic, workload, and human identity estates.

With 80% of organisations already seeing AI agents act beyond intended scope, according to AI Agents: The New Attack Surface report, the next maturity step is to align agent governance with standards such as NIST AI Risk Management Framework and the OWASP Agentic AI Top 10.

For practitioners

• Inventory every agent and owner Create a live register of approved agents, business owners, data access, and permitted use cases. Explicitly document what each agent must not do, because unsupported use is where shadow AI starts. This inventory should tie directly to identity governance and review workflows.
• Assign per-agent identity and least privilege Issue unique identities for each agent and scope access to the smallest set of tools, data, and actions needed for the task. Use the same access review discipline you would apply to any privileged non-human identity, but add runtime traceability and owner accountability.
• Remove destructive tools from the agent path Do not rely on prompt instructions alone to keep an agent safe. Segregate tools, place irreversible actions behind human approval, and enforce a deterministic layer that can block high-risk actions even when the model proposes them.
• Test for reproducibility and drift Re-run the same tasks and compare outputs, tool calls, and downstream effects. If the agent behaves differently across runs, treat that as a control signal, not just a model quirk, and tighten the workflow before extending access.

Key takeaways

• AI agents are already behaving like privileged identities, which puts identity governance at the centre of AI security.
• The main failure is not model quality alone, but the mismatch between machine-speed actions and human-paced control assumptions.
• Per-agent identity, runtime enforcement, and reproducible audit trails are the controls that change agent governance from theory to practice.

Key terms

• Per-instance agent identity: A per-instance agent identity is a distinct credential and governance record assigned to one running agent instance, not to the whole agent type. It allows security teams to revoke, audit, and scope access at the level where action actually occurs, which is essential when agents can spawn or delegate.
• Runtime Enforcement: Runtime enforcement is the practice of blocking malicious behaviour while software is running, rather than only detecting it after the fact. It monitors process activity, network actions, and privilege changes so a live attack can be interrupted at the point of execution.
• Deterministic Logging: Logging that records actions, inputs, outputs, and decisions in a consistent way that can be replayed or audited later. In AI agent governance, deterministic logging is how teams prove what happened when model behaviour is non-deterministic or changes between runs.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.

What's in the full article

FireCompass's full blog post covers the operational detail this analysis intentionally leaves for the source:

• The roundtable framing and participant context behind the governance positions, including the role of Bruce Schneier.
• The practical control set leaders discussed for runtime enforcement, approval gating, and tool segregation.
• FireCompass's own agentic pentesting example, including how its harness uses deterministic controls and critic agents.
• The article's discussion of why reliability can erode human oversight as agents improve.

👉 FireCompass's full post covers the roundtable positions, control set, and agentic pentesting example.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM or identity governance programme, it is worth exploring.