AI risk in generative systems is really identity risk

At a glance

What this is: This is a practitioner analysis of how generative AI risk emerges from semantic manipulation, untrusted context, and excessive tool access.

Why it matters: It matters because AI systems that can act, retrieve, and disclose data create new governance gaps for IAM, PAM, NHI, and agentic AI programmes.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).
• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.

👉 Read Akto's analysis of AI risks in generative systems and agentic workflows

Context

AI risk in generative systems is not just a model-quality problem. It is a governance problem created when natural language, retrieved context, and tool execution share the same operational boundary. In practice, that means prompt injection, data leakage, hallucination, and unsafe tool use can all turn into identity and access failures once an AI system is allowed to read, retrieve, or act on enterprise resources.

The critical issue for IAM and NHI teams is that AI agents blur the line between authenticated user, service account, and delegated automation. When an agent can call APIs, send emails, or query sensitive repositories, the real control question becomes who or what is authorised to act, under which conditions, and how that authority is bounded across the workflow.

Key questions

Q: What breaks when AI agents are given broad enterprise access without tight governance?

A: Broad access turns AI agents into high-speed execution paths that can move data, spend money, modify records, or delete assets before operators can intervene. The failure is not only misuse by an attacker. The system itself can exceed intended scope during normal task completion, so the real control problem is bounding authority before runtime action begins.

Q: Why do AI agents make non-human identity governance harder?

A: AI agents make governance harder because they can request tools, act autonomously, and change behaviour across sessions while still relying on machine credentials. That increases the number of access paths security teams must supervise. The result is a stronger need for task-scoped access, explicit ownership, and continuous monitoring of what the agent can reach.

Q: How do security teams know whether AI access is actually working safely?

A: Look for three signals: complete discovery of the AI estate, clear mapping of source data to each system, and logs that prove what was accessed and why. If any of those are missing, the control environment is incomplete. Safe AI access is evidenced, not assumed.

Q: Who is accountable when an AI agent accesses the wrong data?

A: Accountability sits with the team that defined the agent’s scope, the owner of the delegated user context, and the operators who allowed access to persist beyond the task. For customer workflows, audit logs should show both the agent and the user identity so responsibility can be traced clearly.

Technical breakdown

Why prompt injection is an access-control problem

Prompt injection works because LLMs do not intrinsically separate instructions from data. A malicious prompt, web page, document, or email can reshape model behaviour, especially when the system blends user text, retrieved content, and system instructions in one context window. The failure is not a classic code exploit. It is semantic override, where the model treats hostile input as legitimate direction and then propagates that decision into downstream tools or outputs. Practical implication: treat all untrusted input as data until it is validated at the execution boundary.

Practical implication: validate prompts, retrieved content, and tool outputs separately before any action is taken.

How excessive agency expands the blast radius of AI agents

AI agents become risky when they are given API access, file permissions, or transaction rights that exceed their task scope. Once an agent can act autonomously, a single malicious instruction can trigger a chain of legitimate-looking operations, such as deleting records, changing permissions, or exfiltrating data. This is where identity governance intersects directly with AI security. The core issue is not only model behaviour, but delegated authority without tight scoping, session limits, or approval boundaries. Practical implication: define task-scoped permissions for every agent and revoke standing access wherever possible.

Practical implication: scope agent permissions narrowly and remove standing access wherever the task allows.

Why RAG and training data create hidden disclosure paths

Retrieval-augmented generation and training pipelines can surface information the requester was never meant to see. If indexed content is overexposed, poisoned, or permission mismatched, the model can retrieve and summarise sensitive documents for an unauthorised user. Training data introduces a second problem, because memorised secrets, PII, or internal logic may reappear in outputs. The security pattern is familiar to identity teams: access control is only effective if it is enforced at every layer, including search, retrieval, and generation. Practical implication: align retrieval permissions with source-system entitlements and monitor for data memorisation risk.

Practical implication: enforce source-level entitlements in retrieval systems and monitor for memorised sensitive data.

NHI Mgmt Group analysis

AI risk management is now identity governance by another name. The article correctly shows that the most dangerous GenAI failures are not limited to model inaccuracies. They arise when AI systems are allowed to interpret untrusted context and act through connected tools, which makes authorisation, delegation, and auditability the real control plane. For IAM and PAM teams, the practical conclusion is that AI security cannot be bolted on after deployment.

Excessive agency is the named failure mode behind most AI misuse. When an agent can send email, query databases, or change records without fine-grained scoping, a prompt becomes an instruction path to enterprise action. That is an identity problem because the system has been given privileges without the governance model usually required for human administrators or service accounts. Practitioners should treat agent permissions as ephemeral, task-bound identities, not as generic application entitlements.

Prompt injection exposes a trust boundary that traditional application security does not fully cover. The article shows that AI systems often collapse instruction and data into the same processing layer, which creates control gaps between input filtering, retrieval, and execution. That means secure design has to assume hostile context inside the workflow, not just at the perimeter. The field needs more execution-bound validation and less confidence in model-side safety alone.

Verification trust gap: AI programmes often know what the model said, but not whether the underlying data, prompt, or action was authorised. This gap becomes visible when compliance teams, legal teams, and executives do not share the same view of what agents accessed or disclosed. In identity terms, visibility without governance is not control. Practitioners should align audit, policy, and entitlement evidence before expanding AI access.

Zero trust for AI only works if it is enforced as conditional authorisation, not as a slogan. The article’s mitigation advice points toward least privilege, continuous monitoring, and human approval for critical actions. Those are valid controls, but the broader lesson is that AI governance must constrain every tool call, retrieval step, and output path. The conclusion for security leaders is simple: if an AI system can act, it needs identity-grade controls.

What this signals

Verification trust gap: AI programmes increasingly fail at the point where identity evidence should exist. If teams cannot prove which prompt, retrieval source, or tool invocation led to an action, they do not have governance, only observability fragments. The near-term priority is to make AI actions attributable to a bounded identity and a validated purpose.

For identity teams, the practical signal is that AI security is converging with PAM and NHI governance. Agents that can act across SaaS, internal APIs, and data stores need conditional access, short-lived credentials, and audit trails that survive compliance review. The right benchmark is not whether the model sounded safe, but whether every meaningful action can be authorised and reconstructed.

For practitioners

• Separate instruction, retrieval, and execution controls Inspect every AI workflow for places where natural-language input can influence tool use, database queries, or downstream automation. Enforce independent validation before the model can execute any action, especially where retrieved context comes from external documents or user-generated content.
• Scope every agent to a task-bound identity Assign each AI agent the minimum permissions needed for a single workflow and remove standing access wherever practical. Use short-lived credentials, explicit approval for high-risk actions, and separate identities for retrieval, generation, and execution paths.
• Audit AI access like any other privileged identity Log prompts, retrieved documents, tool calls, and final outputs so you can reconstruct what the agent saw and what it did. Without that evidence, you cannot prove whether the system accessed sensitive data, misused permissions, or crossed an authorisation boundary.
• Map AI workflows to zero trust controls Apply conditional access, least privilege, and step-up approval to AI systems that touch sensitive data or operational tools. Where the workflow can change state, require human confirmation before the action is committed, not after the output is generated.

Key takeaways

• Generative AI risk becomes an identity problem the moment a model can retrieve data, call tools, or act on behalf of a user.
• The most important failure mode is excessive agency, where broad permissions let manipulated agents cross authorisation boundaries.
• Security teams need execution-bound validation, task-scoped identities, and full auditability before AI access expands further.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Excessive agency: A condition where an AI system is given more operational authority than its task requires. The risk is not just poor output. It is that mistakes, manipulation, or compromise can produce destructive actions at machine speed across the systems the agent can reach.
• Retrieval-augmented Generation: Retrieval-augmented generation is a pattern where an AI model pulls external information before generating output. The security challenge is that access rules can weaken when data is chunked, embedded, cached, or reused, so source permissions may not automatically follow the content into the model's context.
• Task-Scoped Identity Boundary: A control boundary that links access, execution, and evidence to one defined task rather than to a persistent user or machine profile. This matters for agentic delivery because the trust model should end when the task ends, not when a person logs out.

What's in the full article

Akto's full blog covers the operational detail this post intentionally leaves for the source:

• Concrete examples of prompt injection, indirect injection, and RAG poisoning in production AI workflows
• Implementation guidance for securing tool calling, API access, and output handling in agentic systems
• Operational controls for logging prompts, model outputs, and retrieval events for investigation and audit
• Mitigation patterns for reducing agent autonomy without breaking business workflows

👉 Akto's full post covers prompt injection paths, RAG exposure, and agent control patterns in more detail

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect access control, lifecycle oversight, and auditability to real identity programmes.