LLM and GenAI security is becoming a distinct governance problem

At a glance

What this is: This is a year-ahead analysis of LLM and GenAI security that argues the category is moving into distinct governance territory, with Shadow AI, visibility, and policy control becoming the main problems.

Why it matters: IAM practitioners need to treat GenAI as an identity and governance issue because model use now intersects with access, data exposure, and accountability across human and non-human workflows.

By the numbers:

• Bard drew around 140 million monthly visitors.

👉 Read Lasso Security's 2024 outlook on LLM and GenAI security trends

Context

Generative AI security is the governance problem that appears when model use, data access, and business workflows converge without clear identity controls. In this article, the primary issue is not model performance but the lack of visibility into who or what is using GenAI tools, what data they can reach, and how those interactions are governed across the enterprise.

Lasso Security frames 2024 as a shift from experimentation to implementation, which means security teams will need to treat LLMs and GenAI tools as part of the access surface rather than a separate innovation layer. That shift affects human use, service integrations, and any emerging agentic behaviour that can act on data or systems without a stable review path.

The article’s starting point is typical for the market, not unusual: adoption is outrunning policy, and security programmes are being asked to catch up after usage has already spread.

Key questions

Q: How should security teams govern Shadow AI in the enterprise?

A: Start by inventorying every place GenAI can be used, including approved tools, embedded assistants, browser access, and API-connected workflows. Then map each use case to data classes, identity owners, logging requirements, and approval paths. Shadow AI becomes manageable only when discovery, policy, and enforcement are connected to the same control model.

Q: Why do GenAI tools create new identity governance problems?

A: GenAI tools can move from isolated experiments into embedded workflows without a clear boundary between the user, the service account, and the system performing the action. That breaks traditional assumptions about static application access, because the effective data exposure and control surface changes at runtime. Identity teams need to govern the workflow, not just the login.

Q: How can organisations know if GenAI policy is actually working?

A: Look for proof that policy is enforced at the point of use: denied sensitive prompts, blocked data transfers, retained audit logs, and documented exceptions. If people can use GenAI freely without traceability or review, policy exists on paper but not in operations. Measurable enforcement is the only reliable signal of control.

Q: Should IAM teams treat GenAI as part of access governance?

A: Yes. GenAI is part of access governance whenever it can read, transform, or disclose enterprise data, because the key question is who or what is authorised to invoke the model and under what conditions. IAM teams should define identity ownership, access scope, logging, and review for each GenAI workflow before adoption scales.

Technical breakdown

Shadow AI and GenAI visibility gaps

Shadow AI describes LLM and GenAI tools in use without central approval, inventory, or governance. The technical problem is not just discovery, but tracing prompts, outputs, and downstream data movement across browsers, SaaS apps, APIs, and embedded assistants. Once GenAI is embedded in workflows, access decisions become distributed across multiple control planes, which makes traditional app inventories too shallow to show actual exposure. For IAM and security teams, the relevant unit is not the model alone but the identity path that reaches it and the data path that leaves it.

Practical implication: build a living inventory of GenAI tools, then map identity, data, and session telemetry to each one.

GenAI-specific policy and evaluation controls

GenAI-specific policy is the layer that defines what data may be sent to models, what logs must be retained, and which use cases require review before deployment. Evaluation frameworks matter because model behaviour can change with prompts, context size, or embedded retrieval sources, so security teams need repeatable checks for data leakage, hallucination risk, and policy bypass. In practice, this is where human governance, NHI controls, and emerging autonomous use cases start to overlap, because the same prompts or connectors may be used by employees, applications, or AI agents.

Practical implication: require pre-deployment evaluation for any GenAI workflow that can touch sensitive data or production systems.

AI regulation and accountability in enterprise workflows

The article points to AI regulation as a forcing function for tighter accountability. That matters because policy without traceability is not enforceable: you need to know which user, system, or integration initiated the model interaction, what data was exposed, and what was produced. In IAM terms, this is an authorization and audit problem as much as a data problem. The closer GenAI gets to decision support or action execution, the more important it becomes to preserve attributable identity boundaries across people, workloads, and machine-mediated access.

Practical implication: align GenAI logging and approval workflows to audit requirements before the use cases expand further.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Shadow AI is the first governance failure, not the last security symptom. The article’s central problem is uncontrolled GenAI adoption before policy and inventory are in place. Once users and applications start embedding LLMs into everyday work, the organisation loses line of sight into data flow, authorisation, and accountability. The practitioner conclusion is simple: if you cannot enumerate the model, you cannot govern the identity path that reaches it.

GenAI security is becoming a distinct control domain because existing IAM assumes stable application boundaries. LLM use breaks that assumption when prompts, plugins, connectors, and retrieval sources shape what data is exposed at runtime. That means access review alone is too shallow, because the effective behaviour of the workflow changes after provisioning. The implication is that GenAI governance must be treated as a runtime control problem, not only a provisioning problem.

Policy without evaluation is only documentation. The article correctly points toward AI evaluation frameworks, NIST AI RMF, CISA guidance, and OWASP controls because GenAI workflows need repeatable checks before they reach production. Security teams cannot rely on adoption enthusiasm or developer intent to compensate for data leakage and misuse risks. The practitioner conclusion is that GenAI policy must be testable, measurable, and tied to enforcement points.

The market is moving toward a three-way governance problem across human, NHI, and emerging autonomous use cases. The same GenAI stack can be used by employees, service accounts, and eventually AI agents, which means identity governance has to follow the actor, not the interface. That is why LLM security is not a niche add-on to cybersecurity. The practitioner conclusion is to design for identity-aware governance before autonomous behaviours make the control gap harder to close.

Named concept: GenAI governance drift. This article shows how fast a programme can move from experimentation to unmanaged operational reliance when policy, telemetry, and accountability lag behind adoption. GenAI governance drift is the gap between visible use and governed use, and it widens whenever the organisation treats AI tooling as a feature instead of an identity surface. The practitioner conclusion is to close that drift before GenAI becomes embedded in core workflows.

From our research:

• 98% of companies plan to deploy even more AI agents within the next 12 months, despite documented rogue behaviour in 80% of current deployments, according to AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
• Read OWASP NHI Top 10 for the control patterns that matter when GenAI workflows start behaving like agents.

What this signals

GenAI governance drift is what happens when adoption outpaces inventory, policy, and enforcement. With 98% of companies planning to deploy more AI agents within 12 months and 80% already seeing rogue behaviour, the next programme failure will not be lack of interest. It will be the inability to trace which identity initiated the model interaction and which data left the boundary.

The practical signal for security leaders is that GenAI can no longer sit only with innovation or product teams. The operating model now needs identity ownership, logging, and legal review for every workflow that can touch sensitive data, especially where NIST AI 600-1 Generative AI Profile controls are expected to support accountability.

As the deployment surface expands, teams should pair discovery with action controls and prepare for agent-like behaviour even in tools that were originally framed as assistants. The more the workflow can read, transform, and disclose data, the more it needs the same scrutiny applied to privileged access paths and external integrations.

For practitioners

• Inventory every GenAI entry point Map browser use, SaaS copilots, embedded assistants, and API-based model access into a single inventory so Shadow AI is visible to security, IAM, and compliance teams.
• Define GenAI data-handling policy Specify which data classes may be sent to models, which require masking, and which are prohibited, then enforce those rules at gateways and application controls.
• Add pre-production evaluation gates Test model-powered workflows for leakage, policy bypass, and unsafe outputs before they reach production, especially where prompts or retrieval sources can change behaviour.
• Tie logging to accountable identities Ensure GenAI actions can be traced back to the initiating user, service account, or application so audits can reconstruct who accessed what and why.
• Align AI governance with security and legal Build approval, retention, and escalation paths that include security, IT, legal, and governance teams so AI use cases do not expand faster than oversight.

Key takeaways

• GenAI security is now an identity and governance problem, not only a model-risk problem.
• Shadow AI, weak visibility, and unenforced policy are the main reasons GenAI adoption becomes operational risk.
• Security teams should inventory, evaluate, and govern GenAI workflows before they become embedded in core business access paths.

Key terms

• Shadow AI: Shadow AI is the use of GenAI tools, models, or assistants without formal approval, inventory, or governance. In practice, it creates an invisible access surface because security teams cannot see what data is being sent, what identities are involved, or what outputs are being reused downstream.
• GenAI governance drift: GenAI governance drift is the gap that forms when AI adoption moves faster than policy, logging, and enforcement. The organisation thinks it has control, but actual use has already spread into workflows that were never reviewed, bounded, or mapped to accountable identities.
• AI evaluation framework: An AI evaluation framework is a repeatable set of tests for model-enabled workflows, usually focused on leakage, policy bypass, harmful outputs, and operational fit. For security teams, it is the bridge between experiment and production because it turns subjective confidence into measurable control.
• Identity-aware GenAI governance: Identity-aware GenAI governance is the practice of tying model usage to the specific user, service account, or application that initiated it. This matters because the same model can behave differently depending on who invokes it, what data is attached, and which systems the workflow can reach.

Deepen your knowledge

GenAI governance drift and Shadow AI control are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building oversight for model-enabled workflows, it is worth exploring.

This post draws on content published by Lasso Security: Wrapping Up 2023, Anticipating LLMs and GenAI Trends in the Year Ahead. Read the original.