Why do LLMs complicate secrets and identity governance?

Why This Matters for Security Teams

LLMs are not just another app tier. They sit at the junction of prompts, retrieval, memory, and tool execution, which means identity boundaries now shape what the model can see and do. If a connector, service account, or API key is too broad, the LLM can turn a small access mistake into a data exposure event. That is why this topic sits squarely in both secrets management and NHI governance.

NHIMG’s Guide to the Secret Sprawl Challenge shows how quickly hidden credentials accumulate across systems, and the same pattern appears when LLMs are wired into enterprise data. Current guidance from the NIST AI Risk Management Framework and OWASP Agentic AI Top 10 treats access paths, tool permissions, and data exposure as core risk surfaces, not implementation details. The practical issue is that LLMs do not simply “read” data. They can retrieve, summarize, transform, and route it across systems that were never designed to share one identity boundary.

In practice, many security teams discover the problem only after an internal assistant returns sensitive material that no ordinary user should have been able to assemble.

How It Works in Practice

The mechanics are straightforward but easy to misjudge. An LLM often runs with a service account, an agent token, or a chain of connector credentials that unlocks email, documents, ticketing, code, or infrastructure tools. If those credentials are static and long-lived, the model inherits standing access even when the task is narrow. That creates an identity mismatch: the user may ask one question, but the model can traverse multiple systems with the same privilege set.

Security teams are increasingly moving toward least privilege, just-in-time access, and short-lived secrets for LLM workloads. The best pattern is to scope identity to the task, not to the application owner’s convenience. That usually means combining workload identity with runtime policy checks, so access is granted only when the request context, tenant, data class, and tool invocation all match approved conditions. For implementation guidance, the OWASP Non-Human Identity Top 10 is useful for credential lifecycle failures, while CSA MAESTRO agentic AI threat modeling framework helps teams reason about tool chaining and action boundaries.

• Use separate identities for the model runtime, each connector, and each downstream tool.
• Prefer ephemeral tokens with explicit TTLs over shared static API keys.
• Restrict retrieval to the minimum data source and classification needed for the task.
• Log tool calls, prompt context, and secret access as one audit trail.

NHIMG’s 12,000 Secrets Found in Public LLM Training Dataset illustrates why secrets exposure cannot be treated as a narrow repository problem. These controls tend to break down when an agent can chain tools across disconnected SaaS platforms because identity context is lost between systems.

Common Variations and Edge Cases

Tighter identity control often increases operational overhead, requiring organisations to balance automation speed against auditability and revocation discipline. That tradeoff is real, especially where LLMs support developers, analysts, or help desks that expect broad search and fast response times.

There is no universal standard for this yet, but current guidance suggests a few edge cases deserve extra caution. First, shared copilots embedded in productivity suites can blur the line between user access and application access, which makes it hard to tell whose privileges were actually used. Second, retrieval-augmented systems may expose data that was not directly requested if the embedding or indexing layer is too permissive. Third, multi-agent workflows can amplify mistakes because one agent’s output becomes another agent’s input, expanding the blast radius of a single over-privileged token.

NHIMG’s 52 NHI Breaches Analysis is a useful reminder that identity failures often begin with convenience, not malice. For broader governance alignment, NIST AI 600-1 Generative AI Profile and NIST Cybersecurity Framework 2.0 both reinforce the need for traceability, access control, and continuous monitoring. In practice, the hardest failures show up where LLMs are allowed to keep working after the original user session, because the identity context no longer matches the action being taken.

Related resources from NHI Mgmt Group

• What is the difference between agent identity governance and secrets management?
• Why do machine identities complicate identity governance more than human accounts?
• Why do legacy identity systems complicate non-human identity governance?
• Why do mergers and acquisitions complicate multi-tenant identity governance?