LLM risks are becoming an IAM problem, not just an appsec issue

At a glance

What this is: This is an analysis of why LLMs create a distinct security category, with the central finding that conventional application controls do not adequately govern model behaviour, data exposure, and access decisions.

Why it matters: It matters because IAM, NHI, and human access programmes now have to govern prompts, outputs, plugins, and AI-driven actions as part of the same identity surface.

By the numbers:

• Gartner predicts that at least 30% of generative AI projects will be abandoned after proof of concept by the end of 2025, with inadequate risk controls cited as a key factor alongside poor data quality, escalating costs, and unclear business value.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).

👉 Read Lasso Security's analysis of LLM risks, identity exposure, and enterprise controls

Context

LLM risks are the security, privacy, and integrity problems that appear when generative models process prompts, data, and tools in ways that are not fully predictable. The key governance gap is that these systems do not fit cleanly into traditional application security, because the attack surface now includes model inputs, outputs, embeddings, plugins, and the identity controls around them.

For IAM and NHI teams, that changes the problem from protecting code paths to governing how model-enabled systems are authorised, observed, and constrained. Human workflows, service credentials, and AI-mediated actions are increasingly intertwined, so the real question is which control planes still assume deterministic behaviour when the workload is probabilistic.

Key questions

Q: How should security teams govern LLMs that can use plugins and APIs?

A: Treat every plugin and API connection as privileged access, not as a feature toggle. Bind each tool to a narrow task, scope credentials to the minimum reachable action, and log every invocation alongside the originating prompt and user context. The control objective is to prevent the model from inheriting more authority than the task requires.

Q: Why do LLMs create risk for IAM and NHI programmes?

A: Because they blur the line between user intent, system behaviour, and delegated access. A model can expose sensitive data, invoke tools, or move information between systems while operating under credentials that were never designed for that style of use. IAM and NHI teams therefore need policy, logging, and revocation controls that fit dynamic model behaviour.

Q: What breaks when organisations trust LLM outputs too much?

A: Downstream systems can treat hallucinated or manipulated output as if it were verified business logic. That can cause data leakage, unsafe decisions, or privileged actions based on text the model generated rather than on an approved source of truth. The failure is not just bad content. It is the absence of validation before execution.

Q: Who should be accountable for LLM security and governance?

A: Accountability should sit with the team that owns the model’s operational behaviour, not only with the team that built it. Security, compliance, data, and business owners all need defined roles for prompts, data access, output review, and incident response. Shared responsibility only works when each control has a named owner.

Technical breakdown

Prompt injection and output abuse in LLM systems

Prompt injection works by inserting malicious instructions into user content, retrieved data, or third-party text so the model treats them as higher priority than the intended task. Output abuse happens when downstream systems trust generated responses too much, allowing hallucinated commands, unsafe transformations, or leakage of secrets to become operational actions. This is not classic code exploitation. The failure is that language becomes an input channel into business logic, and the model cannot reliably separate instruction from data without guardrails around context, retrieval, and response handling.

Practical implication: restrict what the model can read and return, then validate every model output before it reaches another system.

LLM plugin and API integrations expand identity risk

When an LLM can call plugins or APIs, it inherits the privileges of those connected systems. That creates a merged identity layer where the model may access Slack, GitHub, databases, or internal services through delegated credentials. Excessive agency is the dangerous pattern here, because the model can be given more reachable capability than its task actually requires. The architectural issue is not just authentication. It is authorisation scope, contextual policy enforcement, and logging across every tool invocation the model can initiate.

Practical implication: bind each model action to a narrow policy and review every tool integration as if it were a privileged workload account.

Training data, embeddings, and model supply chain exposure

LLM supply chain risk appears when models, adapters, datasets, or vector stores are introduced without full provenance and integrity controls. Poisoned data can quietly alter model behaviour, while tampered embeddings or public repositories can introduce hidden instructions or malicious dependencies. Because these systems learn from what they ingest, corruption is not limited to a single bad file. It can persist into the model’s behaviour and outputs. That makes provenance, reproducibility, and version control part of the security baseline, not just the development workflow.

Practical implication: treat training and retrieval inputs as governed assets and require provenance checks before anything is promoted into production.

Threat narrative

Attacker objective: The attacker aims to turn the model’s legitimate access and trusted outputs into a path for data theft, control bypass, or unauthorized cross-system action.

1. Entry occurs when adversaries inject malicious prompts, poisoned content, or compromised dependencies into the model’s input, retrieval, or plugin path. Credential access follows when the model is induced to expose sensitive data, reusable tokens, or internal context through its own outputs or tool calls. Escalation happens when those outputs are trusted by downstream systems or used to trigger higher-impact actions across connected applications. Impact is data leakage, unauthorized action, or business process disruption at machine speed.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM risk management is becoming an identity governance problem before it is an appsec problem. The article shows that the most consequential failures sit in prompts, outputs, plugins, and data handling rather than in code alone. That shifts control ownership toward IAM, NHI, and policy enforcement teams because the model is now acting across governed resources, not just producing text. Practitioners should treat LLM security as an access and accountability discipline, not a narrow content-safety issue.

Context-based access control is the right named concept for this category, because LLM behaviour depends on what the model can see, not just who launched it. RBAC is too coarse when a model’s risk changes with the prompt, the retrieval set, or the downstream tool chain. The article’s own comparison makes that clear by contrasting context-based controls with static least privilege. Practitioners should reframe model governance around runtime context, not fixed entitlement tables.

Shadow AI creates an ungoverned NHI layer inside the enterprise. When employees use external LLM services without approval, they move sensitive data into systems that may retain prompts, reuse content, or connect to external tools. That is structurally similar to unmanaged service accounts, except the behaviour is conversational and often invisible to security teams. The implication is that discovery and ownership matter as much as policy design.

Excessive agency is the control gap that turns LLM integrations into a lateral-movement path. The article’s agentic example shows what happens when a model is allowed to read, write, and act across multiple systems through shared identity and tool access. That is not a conventional application vulnerability. It is delegated power without containment. Practitioners should recognise that the breach condition is over-broad operational reach, not model accuracy.

LLM governance now has to span human, NHI, and autonomous-like behaviour in one programme. The same system can expose human-entered secrets, exercise non-human credentials, and trigger tool actions that look autonomous from the outside. That convergence means the old split between user access, workload identity, and AI oversight is no longer workable. Practitioners should align policy, logging, and approval boundaries across the full identity chain.

From our research:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%), according to AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
• That governance gap becomes sharper when you compare it with OWASP NHI Top 10, which helps teams translate model behaviour into controls.

What this signals

Context-bound access control is becoming the practical bridge between LLM risk and identity governance. The next wave of programmes will not succeed by bolting model monitoring onto legacy RBAC alone. Teams need to define what the model may see, which tools it may call, and which actions require explicit approval, then align those decisions to the same control ownership used for non-human identities and delegated workload access.

As LLMs spread, the real operational question is which parts of the enterprise still assume deterministic access patterns. With 48% of companies unable to track and audit the data their AI agents access, the blind spot is already measurable. The programme response is to treat prompts, retrieval, and tool use as governed events, not just model telemetry.

Enterprises that already map service accounts, secrets, and privileged workflows have an advantage here because the same discipline now applies to model-driven execution. The difference is that model behaviour is more variable, so discovery, auditability, and approval paths must be tighter, not looser.

For practitioners

• Classify every LLM integration by identity and access pattern Document whether the system is only generating text, reading governed data, or calling tools with delegated credentials. Map those behaviours to the actual identity type in play and require an owner for each path, including shadow AI discovered outside approved workflows.
• Constrain tool-calling models with context-bound policies Limit what the model can read, write, and invoke based on task context, data sensitivity, and session purpose. Review plugin and API access as privileged workload access, not as harmless application extensibility.
• Monitor prompts, outputs, and downstream actions as one audit trail Centralise logs for inputs, model responses, retrieval events, and tool invocations so investigators can reconstruct how data moved and where trust was broken. Without that chain, prompt injection and output abuse remain hard to prove.
• Tighten supply-chain controls around models and embeddings Require provenance checks for datasets, adapters, and vector stores before promotion. Treat tampered training data and public repository imports as intake risks, then revalidate model behaviour after each change.

Key takeaways

• LLM risk is an identity and governance problem because the model can read data, invoke tools, and influence systems in ways traditional AppSec does not cover.
• The article’s own evidence shows that AI behaviour is already exceeding intended scope, which makes governance a current control gap rather than a future concern.
• Practitioners should prioritise context-based access, full audit trails, and supply-chain provenance before expanding model reach further.

Key terms

• Prompt Injection: Prompt injection is the practice of embedding instructions into content that a model reads so the model follows the attacker’s intent instead of the operator’s. In enterprise environments it matters because the malicious text can arrive through documents, web pages, chat, or retrieval sources and then influence output or tool use.
• Context-based Access Control: Context-based access control grants or denies access based on the situation surrounding the request, such as data sensitivity, session purpose, device state, or intended action. For LLMs, the context includes prompts, retrieved sources, and tool calls, which means authorisation has to shift dynamically rather than rely only on static roles.
• Shadow AI: Shadow AI is the use of AI services or models outside approved governance, monitoring, or security controls. It creates invisible data exposure because employees may paste sensitive information into external systems, connect unmanaged tools, or generate outputs that bypass enterprise logging and retention rules.
• Excessive Agency: Excessive agency occurs when an LLM or AI-enabled workflow is given more reachable action than the task needs. The risk is not only inaccurate output. It is the ability to read, write, delete, or trigger changes across systems that should remain outside the model’s operational scope.

Deepen your knowledge

LLM risk management and context-based access control are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are extending governance into model-driven workflows, it is worth exploring.

This post draws on content published by Lasso Security: LLM Risks: Enterprise Threats and How to Secure Them. Read the original.