LLM security needs runtime governance, not policy alone

At a glance

What this is: LLM security is the governance layer that controls how models, copilots, and agents interact with enterprise data and systems, and the key finding is that policy without runtime enforcement leaves material blind spots.

Why it matters: IAM, NHI, and AI governance teams need a common control model because LLMs now touch human users, machine identities, and autonomous execution paths in the same environment.

By the numbers:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.
• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.

👉 Read WitnessAI's full guide to enterprise LLM security and runtime controls

Context

LLM security is the discipline of controlling how large language models, copilots, and autonomous agents interact with enterprise data, users, and systems. The core problem is that traditional security stacks were built for deterministic software, while LLM behaviour is probabilistic, conversational, and increasingly capable of taking action.

That creates an identity governance problem as much as a data protection problem. Human users can paste sensitive information into third-party tools, service accounts can be invoked through AI workflows, and autonomous agents can initiate actions that outpace review cycles and static policies.

The practical challenge is not whether organisations have AI policies. It is whether those policies are enforced with discovery, runtime inspection, data controls, and auditability across the full AI operating surface. That starting point is now typical, not exceptional, in enterprise AI adoption.

Key questions

Q: How should security teams implement LLM security across copilots and agents?

A: Security teams should treat LLM security as a layered control problem, not a model tuning problem. Start with discovery, then enforce runtime inspection, tokenization, and audit trails across all AI touchpoints. The goal is to control what enters the model, what leaves it, and what actions an agent can trigger in downstream systems.

Q: Why do LLMs create new identity and access risks for enterprises?

A: LLMs create identity and access risks because they sit between users and systems while handling data, instructions, and actions in the same workflow. Once an agent can call tools or use connected credentials, the model can move from generating text to exercising delegated access. That changes oversight from content review to access governance.

Q: What do organisations get wrong about AI policy enforcement?

A: The common mistake is assuming that a written policy meaningfully constrains AI behaviour without technical enforcement. Policies do not stop data leakage, prompt injection, or agent tool misuse unless they are applied at runtime. Effective governance requires controls that are measurable, repeatable, and auditable in production.

Q: How can teams tell whether AI governance is actually working?

A: Look for evidence of enforced controls, not just policy documents. Useful signals include complete AI inventory coverage, bidirectional inspection logs, tokenization before external model calls, and audit trails that tie agent actions back to the human initiator. If those artefacts are missing, governance is incomplete.

Technical breakdown

Why probabilistic model behaviour breaks traditional security validation

Traditional application security assumes fixed code paths, repeatable outputs, and policy decisions that can be tested against known states. LLMs do not behave that way. The same input can produce different outputs, and the model may follow natural-language instructions that were never intended as authoritative commands. That makes validation harder because security teams cannot rely on static test cases to prove the model will behave safely in every context. The control problem is therefore external to the model: governance has to sit around inference, not inside it.

Practical implication: security teams need controls that inspect prompts, outputs, and policy context at runtime, not just pre-deployment model testing.

How prompt injection turns language into an attack vector

Prompt injection works because the model treats natural language as executable instruction content rather than as untrusted text. A malicious instruction can be buried in an email, document, webpage, or uploaded file, then activated when the model ingests that content during a legitimate task. The danger is not only data leakage. The model may also alter its behaviour, expose hidden instructions, or comply with a command that was never authored by the user. This is why content inspection has to extend to both inputs and outputs.

Practical implication: organisations should inspect model inputs from documents, web content, and user prompts before they reach the model.

What agentic workflows change about LLM security and access control

An LLM becomes materially more dangerous when it can call tools, query systems, or execute multi-step workflows. At that point, the model is no longer only generating text. It is participating in action chains that can trigger business processes, access data stores, and modify records. In identity terms, the risk shifts from content misuse to delegated execution, where the model’s output can become an authenticated action through connected tools and tokens. That is why agent guardrails, tool-call inspection, and immutable audit trails matter more than generic chatbot moderation.

Practical implication: treat every agent tool connection as a governed access path and review who can authorise it, monitor it, and revoke it.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM security is now an identity governance problem, not just a content safety problem. The article shows that LLMs sit between people, data, and systems, which means they inherit access, generate actions, and sometimes amplify existing privilege paths. That changes the control objective from blocking bad text to governing who or what can act through the model. Practitioners should treat AI interaction paths as identity-bearing workflows, not as isolated application features.

Runtime enforcement is the missing control plane for enterprise AI. The article’s real point is that policies written on paper do not constrain probabilistic systems unless they are enforced at the moment of prompt submission, model response, and tool execution. That gap is now common across enterprises adopting copilots and agents faster than their governance model can mature. The implication is that enforcement has to move into the execution path, where the risk actually occurs.

Agentic AI creates a delegated action problem that traditional IAM was never designed to review. Access review processes were built for stable entitlements, but agents can initiate actions, chain tools, and complete work across multiple systems without a human clicking every step. That makes accountability and auditability more fragile, especially when human initiation and machine execution blur together. Practitioners should redesign oversight for action chains, not just for login events.

Discovery and visibility are the named concept that separates AI governance from AI hope. The article makes clear that organisations cannot enforce policy across tools they do not know about, especially when AI appears in browsers, native apps, IDEs, and embedded copilots. Without a continuously updated inventory of models, agents, and connections, governance becomes partial by design. The practitioner conclusion is simple: discovery is the prerequisite control for every other AI security decision.

LLM governance now has to span human, NHI, and autonomous identities in one model. The same deployment can involve employees entering data, service accounts backing integrations, and agents making runtime decisions. That cross-domain overlap is where many programmes break, because each control family assumes a different type of actor. The implication is that identity teams need one governance framework that can separate the human prompt, the machine credential, and the autonomous action path.

From our research:

• 98% of companies plan to deploy even more AI agents within the next 12 months, despite documented rogue behaviour in 80% of current deployments, according to AI Agents: The New Attack Surface report.
• Only 44% have implemented any policies to govern AI agents, even though 92% say the issue is critical to enterprise security.
• That gap makes the next governance question operational, not theoretical. Review OWASP NHI Top 10 for the control patterns that map most directly to agentic risk.

What this signals

Discovery will become the first control layer for enterprise AI governance. As copilots and agents spread into browsers, native apps, and developer environments, teams will need a live inventory of AI touchpoints before they can enforce any policy with confidence. The governance model is shifting from “approve the platform” to “understand every path by which AI touches enterprise data.”

AI auditability will be judged by evidence, not intent. Identity teams should expect more scrutiny over whether they can tie prompts, outputs, and agent actions back to users and systems in a way that stands up in investigation or compliance review. That aligns with the direction of NIST AI Risk Management Framework thinking: governance has to be demonstrable in operation, not only documented in policy.

Runtime controls will need to separate human intent from machine execution. When a person starts the workflow but an agent completes the action, the control model has to preserve attribution across the delegation chain. That is where LLM security, NHI governance, and access oversight converge into one programme requirement.

For practitioners

• Inventory every AI touchpoint Build a continuously updated catalogue of chatbots, copilots, embedded AI, agents, and MCP server connections, including where they ingest data and which identities they can act through. Discovery has to include browser-based and native application usage, not just sanctioned platforms.
• Inspect prompts and responses at runtime Enforce bidirectional inspection before prompts reach the model and before outputs reach users or downstream systems. Use policy actions that reflect context, data sensitivity, and intended purpose rather than simple allow or block decisions.
• Tokenize sensitive data before model exposure Replace credentials, PII, customer records, and proprietary values with non-reversible stand-ins before they leave the enterprise boundary. That reduces the chance that third-party models or downstream connectors ever see the original data.
• Treat agent tool calls as governed access paths Review every API, database, and workflow connection an agent can use, and define who can authorise, monitor, and revoke each path. Audit trails should capture the human initiator, the agent action, and the policy decision taken.
• Align AI controls to identity governance reviews Bring AI usage, model access, and agent actions into existing access review and audit processes so the programme can evidence control operation. This is where governance becomes defensible rather than aspirational.

Key takeaways

• LLM security is fundamentally a governance issue because the model now sits inside identity, data, and action paths at the same time.
• The main evidence point is not model sophistication but control coverage: organisations are moving faster than their runtime enforcement and audit capabilities.
• Teams should focus on discovery, runtime inspection, tokenization, and auditability if they want AI controls that hold up in production.

Key terms

• LLM security: LLM security is the set of controls used to manage how language models interact with enterprise data, people, and systems. It combines data protection, policy enforcement, runtime inspection, and auditability so AI can be used without turning every conversation into an uncontrolled access path.
• Prompt injection: Prompt injection is a technique that manipulates a model through text that appears legitimate but contains hidden instructions. In enterprise settings it becomes a control problem because the model may treat untrusted content as authority and act on it during a normal workflow.
• Agentic workflow: An agentic workflow is a process in which an AI system can select tools, access systems, and carry out multi-step actions on behalf of a user or business function. That makes the workflow an identity and access issue, not just a content generation issue.
• Runtime defense: Runtime defense is the enforcement layer that inspects inputs, outputs, and actions while an AI system is operating. It matters because static policy and pre-deployment testing cannot fully control probabilistic behaviour once a model is interacting with live enterprise data.

Deepen your knowledge

LLM security, runtime defense, and AI governance are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building control coverage for copilots and agents from a similar starting point, it is worth exploring.

This post draws on content published by WitnessAI: enterprise LLM security and runtime defense guidance. Read the original.