AI risk assessment frameworks are now identity governance work

At a glance

What this is: A guide to building an AI risk assessment framework for LLMs and agents, with the central finding that autonomous tool use turns AI security into an identity and governance problem.

Why it matters: IAM, IGA, PAM, and security teams need a shared assessment model because agentic systems expand access, amplify blast radius, and create governance gaps that traditional software review does not cover.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.
• 72% of organisations have experienced or suspect they have experienced a breach of non-human identities, 46% confirmed and 26% suspected.

👉 Read Akto's guide to building an AI risk assessment framework for agents and LLMs

Context

AI risk assessment is the discipline of inventorying AI systems, classifying their exposure, testing their threats, and revisiting those judgments as the system changes. In this article, the core issue is not whether a model can answer questions, but whether models, agents, and tool integrations now function as governed identities inside the enterprise.

That matters because agentic AI changes the control problem. Once a system can call tools, chain actions, and operate with standing credentials or shared access, the security team is no longer assessing software output alone. It is assessing runtime authority, scope, and accountability across AI risk assessment, agent identity, and connected data paths.

Key questions

Q: How should security teams test AI agents that can call tools and APIs?

A: They should test the agent's actual execution paths, not just its text responses. That means covering tool calls, protocol handling, approvals, and state transitions with adversarial scenarios that mirror real misuse. If the test cannot reach the access point where the agent acts, it does not meaningfully assess the security risk.

Q: Why do chat-based AI systems create new identity risk for organisations?

A: Chat-based AI systems compress intent, delegation, and execution into one interaction, which makes privilege boundaries easier to cross without noticing. The risk is not only misuse of the model, but also over-scoped connectors, weak approval design, and unclear ownership of actions that begin in conversation but end in external systems.

Q: What breaks when AI risk reviews are done only at deployment time?

A: You miss the behaviour that appears after launch. Models change, prompts are edited, connectors are added, and a previously acceptable workflow can become risky without any formal re-review. Static assessment creates stale assurance, which is especially dangerous when the system can act faster than a human can intervene.

Q: Who is accountable when an AI agent causes a security incident?

A: Accountability should sit with the business owner, the system owner, and the security function together, because agent behaviour crosses operational boundaries. Organisations need a defined owner for approval, monitoring, and retirement, plus audit evidence that shows what the agent accessed and why.

Technical breakdown

Why AI risk assessment differs from traditional software review

Traditional software risk review assumes the system behaves deterministically and that a control remains stable between assessment cycles. AI systems break that assumption because the same input can produce different outputs, and agentic systems can move from text generation to action execution. That means the risk unit is not just the model, but the full chain of prompt, retrieved context, tool call, and downstream effect. In practice, the assessment has to follow behaviour at runtime, not just configuration at design time.

Practical implication: inventory the full action chain, not only the model endpoint.

Why agent permissions create the largest governance gap

Agents become risky when they are granted general credentials to act flexibly, because one manipulated decision can become broad access across systems. The problem is not only privilege size, but privilege duration and reuse across tools. Shared service accounts, standing access, and over-scoped tokens turn a single prompt or tool response into a control-plane event. For IAM and PAM teams, this is the point where AI risk becomes identity governance, because access policy now determines blast radius.

Practical implication: review every agent as an identity with explicit scope, duration, and ownership.

How runtime monitoring closes the gap left by static assessment

Static review cannot see behaviour that only appears in production, such as loops, unauthorized tool calls, or repeated downstream actions. That is why continuous monitoring matters for AI systems that change prompts, models, and integrations after approval. The assessment artefact must stay live, because a closed risk can reopen as soon as a new connector, data source, or agent workflow is added. For security teams, the control objective is not periodic reassurance, but ongoing evidence that behaviour still matches the approved scope.

Practical implication: pair initial scoring with continuous runtime reassessment and change-triggered review.

Threat narrative

Attacker objective: The attacker wants to turn a governed AI workflow into a high-reach execution path that can expose data, misuse tools, or trigger harmful actions at scale.

1. Entry occurs when an AI system is connected to external tools, data sources, or shared credentials that expand its effective authority.
2. Escalation happens when the model or agent is manipulated into over-scoped actions, unauthorized tool use, or repeated execution beyond its intended task.
3. Impact follows when that misplaced authority reaches databases, email, payment systems, or other downstream services and causes data exposure or business harm.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI risk assessment has become identity governance for agentic systems. Once a model can choose tools, act on data, and trigger downstream systems, the security question stops being limited to output quality. The governance question becomes who or what is allowed to act, for how long, and with what blast radius. Security teams that still treat AI review as a one-time model exercise are missing the identity layer that now defines operational risk.

Standing access is the wrong default for autonomous behaviour. Traditional IAM assumes access can be evaluated against a stable task and a stable operator. That assumption fails when an agent can initiate execution, chain actions, and change scope mid-session. The implication is not merely more controls, but a different view of what privilege means when the actor is runtime-driven rather than human-paced.

Ephemeral credential trust debt: Agent programmes accumulate hidden risk when temporary access is granted faster than ownership, review, and revocation can keep up. The more tools and connectors an agent gains, the more the organisation inherits unresolved trust decisions from earlier design choices. That debt becomes visible only when an incident, audit, or model change forces the team to explain why the access was ever acceptable.

Runtime evidence now matters more than policy intent. Risk scoring without runtime validation produces confidence, not assurance. If an agent can behave differently after deployment, then the programme needs evidence of actual tool use, actual data access, and actual escalation paths. Practitioners should treat assessment records as live governance artefacts, not static documentation.

AI governance must align security, compliance, and ownership around the same inventory. A useful framework does more than list risks. It gives auditors, security teams, and product owners a shared view of models, agents, prompts, integrations, and access paths. That is the minimum condition for deciding which AI systems can remain in production and which need tighter scope.

From our research:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials, according to AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
• For a broader control lens, see OWASP Agentic AI Top 10 and NIST AI Risk Management Framework for how assessment should connect to governance.

What this signals

Ephemeral credential trust debt: AI programmes that rely on temporary access still accumulate unmanaged risk if ownership, review, and revocation do not keep pace with deployment. In practice, that means the inventory has to include who can change each agent, who can disable it, and which downstream systems it can reach before the next review cycle. The strongest signal of maturity is not the number of models tracked, but whether access scope is actually tied to runtime behaviour.

With 80% of organisations already reporting AI agents acting beyond intended scope, the governance gap is no longer speculative, and IAM teams should expect agent oversight to move into the same operational lane as privileged access and service account control. The programme question is whether your current review model can see tool use, data access, and change events before those behaviours become an incident.

For teams building a policy baseline, pair internal review with external guidance such as the NIST Cybersecurity Framework 2.0 so AI assessment lands inside govern, identify, protect, detect, respond, and recover rather than sitting in a separate AI-only silo.

For practitioners

• Inventory AI systems as governed identities Track models, agents, vector stores, prompts, templates, and tool integrations across cloud, on-prem, and employee devices. Include shadow AI so your inventory reflects actual runtime exposure rather than approved architecture diagrams.
• Classify AI assets by autonomy, sensitivity, and exposure Use separate risk tiers for internal assistants, public-facing agents, and systems touching regulated data. Tie each tier to review cadence and approval authority so higher-risk agents receive deeper assessment before expansion.
• Scope agent credentials to the minimum viable task Replace shared credentials and standing access where possible, and define explicit permission boundaries for each agent. Review duration, tool reach, and ownership together so one compromised decision does not become a broad access event.
• Validate controls against real agent behaviour Retest input filtering, output validation, permission scoping, and sandboxing after each material change. If a guardrail fails in production, treat that as a governance defect rather than a tuning issue.
• Wire reassessment to change events Trigger reassessment when models, prompts, tools, or data sources change, not only on a quarterly schedule. Continuous monitoring should look for loops, unauthorized tool calls, and new integrations that expand blast radius.

Key takeaways

• AI risk assessment now belongs in the identity governance stack because agentic systems act through tools, credentials, and downstream services.
• The strongest evidence in the field shows agent behaviour already exceeds intended scope in most organisations, which makes static review insufficient.
• Programmes that inventory access, validate controls at runtime, and trigger reassessment on change will be better positioned to govern AI safely.

Key terms

• AI Risk Governance: AI risk governance is the operating model for deciding who owns AI policy, how controls are enforced, and how exceptions are handled. It connects security, data, and operational teams so AI behaviour is managed as part of the wider identity and control stack.
• Agentic AI: Autonomous AI systems capable of planning, deciding, and taking actions — including calling APIs, writing code, and orchestrating other agents — with minimal human oversight. Agentic AI introduces new NHI risks as agents must authenticate to external services.
• Runtime Monitoring: The practice of observing identity activity while it is happening, not after the fact. For agents and NHIs, it means tracking tool calls, credential use, and resource access in real time so deviations from approved scope can be detected before damage compounds.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.

What's in the full article

Akto's full blog post covers the operational detail this post intentionally leaves for the source:

• Step-by-step guidance for discovering AI systems, agents, models, and tool integrations across mixed environments.
• Detailed risk scoring workflow for prioritising AI findings by likelihood, impact, and compliance exposure.
• Implementation examples for permission scoping, output validation, sandboxing, and runtime reassessment.
• A structured walkthrough of AI risk categories including prompt injection, autonomous action, MCP risk, and supply chain exposure.

👉 Akto's full guide covers the assessment workflow, control validation, and continuous reassessment in more detail.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.