Enterprise LLM security is now a data governance problem

At a glance

What this is: This is a checklist-style analysis of enterprise LLM security, with the central finding that governance gaps, shadow AI, and weak data lineage are exposing sensitive information across prompts, outputs, and agent workflows.

Why it matters: It matters because IAM, PAM, and data security teams now have to govern not just human access to AI tools, but also agentic workflows, third-party integrations, and the identity-bound paths by which sensitive data reaches and leaves models.

By the numbers:

• Cyberhaven Labs research found that 39.7% of the data employees share with AI tools is sensitive.
• Endpoint-based AI agents grew 509% in 2025.

👉 Read Cyberhaven's enterprise LLM security checklist for AI governance and data protection

Context

Enterprise LLM security is the governance layer around who can use large language models, what data they can reach, and how their inputs and outputs are monitored. The core problem is that many organisations are approving AI use faster than they can define access boundaries, lineage controls, and accountability for the data those systems touch.

The identity angle is real, even in a broader AI security discussion. When employees use personal accounts, when agents call internal APIs, and when MCP-connected tools reach enterprise data, the question becomes who or what is authorised to move sensitive information, and how that authorisation is enforced across human, NHI, and agentic workflows.

Cyberhaven's starting position is not unusual. Most enterprises are seeing the same pattern: rapid AI adoption, limited inventory of sanctioned tools, and control frameworks that were designed for file movement rather than dynamic prompt, output, and agent activity.

Key questions

Q: How should security teams implement LLM governance without slowing adoption?

A: Start by governing the model’s access path, not just the application wrapper. Limit what the LLM can read, what it can output, and what actions it can initiate. Then add runtime controls for prompt filtering, data loss prevention, and approval gates so adoption can continue without turning every session into an uncontrolled trust decision.

Q: Why do LLM applications create governance problems for IAM and security teams?

A: LLM applications create governance problems because they can turn untrusted input into live system behaviour. Once a model can call tools, reach internal data, or shape business logic, access control, output handling, and runtime policy become one control surface. That requires identity and application governance to be designed together.

Q: What breaks when traditional DLP is used alone for AI security?

A: Traditional DLP misses much of the risk because prompts, browser submissions, and generated outputs do not always look like file transfers. It also struggles with inference risk, where harmless inputs combine into a sensitive result. AI security needs lineage, context, and identity-aware controls, not only pattern matching.

Q: Who is accountable when sensitive data is retained in a third-party AI tool?

A: Accountability sits with the organisation that allowed the data into the tool, even if the provider stores or processes it. Teams need clear ownership for prompt retention, deletion requests, and vendor data processing terms. If the provider cannot prove erasure or lineage, the organisation still carries the compliance and privacy risk.

Technical breakdown

Shadow AI, sanctioned tools, and discovery gaps

Shadow AI is the use of AI tools or agents outside formal approval paths. The operational problem is not only unsanctioned software, but also sanctioned tools that expand through browser extensions, plugins, and agentic integrations faster than security teams can catalogue them. Discovery matters because every untracked model connection creates a new data path that bypasses policy decisions made at procurement time. In practice, continuous inventory is the only way to know which tools are trusted, tolerated, or restricted.

Practical implication: maintain a live inventory of all AI tools, agents, plugins, and MCP servers before approving data access.

Why traditional DLP misses prompt and output risk

Traditional DLP was built for files, email, and known transfer points. LLM interactions often involve pasted text, browser submissions, generated summaries, and agent actions that do not resemble a classic exfiltration event. That is why data lineage has become more important than content pattern matching alone. A prompt can contain copied confidential material, and an output can reveal proprietary context even when neither object looks sensitive in isolation.

Practical implication: extend controls to the endpoint and the output layer, not just the network perimeter.

Model, third-party, and agent risk in connected AI systems

An enterprise LLM rarely operates alone. It may call external APIs, connect to internal systems, or run through agent frameworks that can read and act on enterprise data. This changes the security model because the model is no longer just generating text. It becomes a decision layer with delegated reach. Governance therefore has to cover provider retention terms, plugin scope, and the permissions granted to any AI system that can chain actions across services. The rise of MCP makes this intersection with identity governance especially important.

Practical implication: treat every model connection as an access pathway that needs review, scope limits, and rollback procedures.

Threat narrative

Attacker objective: The attacker objective is to obtain sensitive enterprise data through AI workflows that were not governed as access-controlled identity paths.

1. Entry occurs when employees use shadow AI, personal accounts, or unapproved integrations to move enterprise data into LLM workflows.
2. Credential access and escalation happen when connected tools, agents, or plugins inherit excessive permissions or reach internal APIs without sufficient review.
3. Impact follows when sensitive data appears in prompts, combined outputs, or agent actions that legacy controls cannot reliably detect or reconstruct.

NHI Mgmt Group analysis

Enterprise LLM security is becoming an identity and data lineage problem, not just an AI tooling problem. The article correctly frames prompt, output, and agent activity as distinct control points, which is where many programmes still fail. When access is mediated by humans, service accounts, plugins, and MCP-connected tools, the governance question is who can move data, under what authority, and with what audit trail. That means LLM security has to sit alongside IAM, PAM, and NHI governance, not outside them.

Shadow AI creates a governance blind spot because approval and usage are no longer aligned. Security teams can only govern what they can see, and the article's discussion of personal accounts and endpoint-based AI agents shows why inventory is now the first control. This is the named concept that matters here: LLM control drift, where policy is written for approved tools but data movement happens through unapproved paths. Practitioners should treat that drift as an access governance failure, not a user training issue.

Data protection for LLMs must shift from static classification to contextual enforcement. A prompt is not a file, and an output can be risky even when it contains no obvious sensitive pattern. That means control decisions need to happen at the point of use, with telemetry tied back to the identity or agent that initiated the action. For identity teams, this reinforces that entitlement scope and lineage are inseparable when AI systems can read, synthesise, and redistribute enterprise data.

Third-party model risk is really delegated-access risk in another form. Once an LLM connects to external providers, APIs, or agent frameworks, the enterprise has created a new trust boundary that behaves like a chain of temporary non-human identities. The question is not whether the tool is clever, but whether its access can be bounded, monitored, and revoked with the same discipline used for other privileged workloads. That is now a core identity governance requirement.

Incident response for AI needs a separate reconstruction model because the evidence trail is different. The article's emphasis on logging, lineage, and playbooks is directionally right, but the field still underestimates how often the first sign of exposure will be a prompt, an output, or an agent call rather than a classic exfiltration alert. Teams need to assume that any AI workflow with data access can become a breach path, and they should design containment around that assumption.

What this signals

LLM control drift: the fastest-growing risk is not model failure, but governance drift between approved AI use and actual data movement. Organisations need to assume that prompts, outputs, and agent actions will cross policy boundaries unless they are instrumented at the endpoint and tied back to identity. For practitioners, that means the next control debate is about lineage, ownership, and revocation, not just acceptable-use language.

The programme-level signal is that AI security is converging with IAM and NHI governance. Once AI systems can call tools, read internal data, and act with delegated authority, they behave like privileged non-human identities with a broader blast radius than traditional software. Teams should align this work with the OWASP Agentic AI Top 10 and NIST AI 600-1 Generative AI Profile, then map access paths back to the identities that created them.

For practitioners

• Build a continuous AI inventory Track every sanctioned and unsanctioned LLM, browser assistant, plugin, agent, and MCP server. Reconcile that inventory against data access approvals so a tool cannot reach production data simply because it was installed by a user or business unit.
• Apply least privilege to AI-connected identities Review the permissions granted to API keys, service accounts, and agent credentials used by AI workflows. Remove broad system access, separate read from write paths, and require explicit approval before a tool can connect to internal data sources.
• Enforce data controls at the point of use Extend controls beyond file DLP to cover pasted prompts, browser submissions, generated outputs, and agent actions. Use endpoint-level visibility and data lineage so sensitive content is tracked even when it never becomes a traditional file transfer.
• Review third-party model terms before deployment Document retention, training, residency, and subprocessor terms for every model provider and connected plugin. Refuse deployments that cannot clearly explain what data is retained and which external systems the model can reach.
• Test an AI-specific incident playbook Run containment exercises for shadow AI, prompt injection, and agent misuse. The playbook should cover tool isolation, evidence preservation, and revocation of the credentials or integrations that allowed the workflow to operate.

Key takeaways

• Enterprise LLM security is really about controlling how people and agents move sensitive data through AI systems.
• The scale of exposure is already measurable, with 39.7% of data shared with AI tools classified as sensitive in Cyberhaven Labs research.
• The practical response is continuous inventory, least privilege for AI-connected identities, and data lineage at the point of use.

Key terms

• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• Data Lineage: The record of how data moves across systems, applications, and workflows. In security operations, lineage shows where sensitive data propagates, which identities touch it, and how a compromise could spread across connected environments.
• Inference Risk: Inference risk is exposure created when a model combines benign inputs into a sensitive output. Unlike classic data leakage, no single input needs to be labelled confidential, so classification tools can miss the problem unless output behaviour is monitored.
• MCP Server: An MCP server is a tool endpoint that connects an AI agent to external systems and data sources through Model Context Protocol. Because it extends what the agent can reach, it becomes part of the identity and access surface and must be reviewed like any other privileged connector.

What's in the full article

Cyberhaven's full article covers the operational detail this post intentionally leaves for the source:

• The six-part enterprise LLM security checklist with specific audit points for discovery, access, data protection, monitoring, and incident response.
• Detailed guidance on how Cyberhaven maps data lineage across prompts, outputs, and agentic workflows.
• Examples of how the vendor classifies sanctioned, tolerated, and restricted AI tools in practice.
• The article's explanation of how its continuous Data Lineage graph supports enforcement decisions.

👉 Cyberhaven's full post covers the checklist details, lifecycle controls, and deployment guidance.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and agentic AI identity. It gives security and identity practitioners a common language for governing delegated access across modern enterprise systems.