Why LLM security breaks when policy stops at deployment

At a glance

What this is: This is an analysis of why LLM security best practices fail in practice and how runtime enforcement changes the control model.

Why it matters: It matters because IAM and security teams now have to govern AI artifacts, agent permissions and runtime decisions as one control problem, not as separate review steps.

Context

LLM security becomes brittle when governance is split across model registries, CI checks, runtime tools and separate team ownership. The first failure is not usually a missing policy, but the absence of a single enforceable control point for the AI artifact itself, especially when agents keep acting after deployment. This is the core design problem behind many LLM security programmes.

The issue also reaches into identity and access control because AI systems do not remain passive once deployed. They authenticate, call tools, read data and trigger actions, so the question is no longer only whether a model was approved, but whether the runtime identity behind it is constrained throughout its lifecycle.

Key questions

Q: Why do LLM security policies break down between teams?

A: They break down because AI systems are assembled from multiple artifacts and owned by multiple teams, so no single review point sees the whole risk picture. When approval stops at deployment, runtime agent behaviour, tool use and later configuration changes escape consistent governance. The fix is to govern the full artifact and enforce policy where the system actually acts.

Q: How should security teams govern LLM and agent access in production?

A: Treat every model, agent, connector, and service account as part of one identity surface. Define owners, scope permissions to task and environment, and require logging that shows who or what accessed the system, what data was touched, and what downstream actions occurred.

Q: Why are AI gateways not enough for LLM security?

A: Gateways help with network traffic, but they do not see every important action. Local file operations, shell commands and stdio-based MCP servers can bypass the gateway boundary entirely. A practical control model needs both network inspection and local execution governance, otherwise the most sensitive behaviour remains outside policy.

Q: What does good evidence collection look like for AI governance?

A: Good evidence collection creates a single audit trail that ties together what was approved, what was allowed at runtime, and what was blocked or redacted. That record should be available to the security team without reconstruction across multiple owners. If an incident occurs, the audit trail should answer who approved the artifact and what the agent did.

Technical breakdown

Why scattered AI artifacts defeat consistent governance

An LLM application is rarely a single deployable unit. It often includes weights, prompts, retrieval data, tool definitions, skill files, MCP servers and configuration, each stored and changed in different places. That fragmentation breaks classic approval flows because reviewers can sign off one component while another changes later. In security terms, the control boundary is too narrow for the system being governed. The practical consequence is that policy has to attach to the artifact bundle, not just to the model binary or the deployment pipeline. See the AI Supply Chain Security and AI-BOM Guide.

Practical implication: Treat the full AI bundle as the governance object, not the model alone.

Why deploy-time checks miss runtime agent behaviour

A deploy gate can validate what was shipped, but it cannot govern what an agent decides to do later. That matters because LLM agents can be manipulated by prompt injection, misuse legitimate tools, or take actions with excessive agency long after approval. The security gap is temporal: the decision surface continues after deployment. Runtime policy enforcement must therefore evaluate each action as it happens, rather than assuming approval at release time is enough. See the Agentic AI Security Guide.

Practical implication: Shift control from pre-deployment approval to policy enforcement at the moment of action.

Why gateways alone do not cover local and non-network actions

An AI gateway is useful for traffic inspection, but it only sees what crosses the network boundary. It does not verify the origin of an agent, and it cannot observe local file operations, shell commands or stdio-based MCP servers that act on the host itself. That leaves a blind spot between network control and endpoint execution. Good governance needs visibility into both the transport path and the local execution path, or else the most sensitive actions stay outside the policy surface. See the MCP Security Guide.

Practical implication: Extend control beyond API traffic to the local execution context where agents actually act.

NHI Mgmt Group analysis

AI governance fails when the control object is fragmented. The post-deployment model review is too narrow for systems that include prompts, tools, data, MCP servers and agent logic. Security teams cannot govern what they cannot version, scan and re-approve as one bundle. The operating assumption that each component can be reviewed independently is now the weakness, not the safeguard. Practitioners should treat the AI artifact as the unit of control.

Runtime is where LLM security becomes real. Approval at build time does not address prompt injection, tool misuse or excessive agency once the system is live. That means governance has to move from static review to continuous action policy, with decisions logged and enforceable at execution time. The field is moving toward runtime control planes because deployment-only controls leave the most dangerous behaviour untouched. Practitioners should expect policy to follow the agent, not the release ticket.

Model governance and identity governance are converging around agent authority. Once an AI system can authenticate, call tools and act on data, it becomes an access subject as much as a model subject. That creates a direct intersection with non-human identity authentication because the real issue is not just what the model says, but what identity and permission boundary it operates under. Teams that still separate AI review from access governance will miss the practical risk surface.

One audit trail is now a governance requirement, not an operational nice-to-have. When models, platform teams, data science and application owners each hold part of the evidence, investigations become fragmented and slow. A durable control plane needs to capture what was approved, what ran, and what actions were blocked or redacted. The practitioner takeaway is simple: if you cannot reconstruct runtime decisions, you cannot govern agent behaviour.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• 24,008 unique secrets were exposed in MCP configuration files in 2025 alone, the protocol's first year of widespread adoption, according to the State of Secrets Sprawl 2026.
• Read next: AI Infrastructure Workload Identity Guide

What this signals

Artifact-first governance is likely to become the default pattern for teams trying to secure LLMs without slowing delivery. The important shift is that policy attaches to the AI bundle and follows it through intake, deployment and runtime, instead of living only in documentation or pipeline gates.

As agent use expands, teams will need to treat runtime authority as a security boundary in the same way they treat credentials and access rights. That means the question is no longer whether an AI system was reviewed, but whether its live permissions and actions stay inside the intended control plane.

For practitioners

• Package AI artifacts as a single governed unit Bundle models, prompts, retrieval assets, tool definitions, agent configuration and related metadata so the security team reviews one versioned object instead of scattered parts.
• Create one intake path for external AI components Force third-party models and components through a private registry, then scan and re-approve them before they reach development, test or production paths.
• Enforce policy at runtime, not just at release Apply the same security rule when an artifact is pulled, admitted to a cluster and used by an agent so runtime decisions cannot outrun governance.
• Log agent actions into the security monitoring path Record approvals, denials, redactions and tool use in a central audit trail that your SOC can investigate alongside other security events.

Key takeaways

• LLM security fails most often because governance is split across artifacts, teams and control points that never see the whole system.
• The critical control gap is runtime behaviour, where agents continue making decisions after deployment approval has already ended.
• Security teams need one governed AI bundle, one intake path and one runtime policy layer if they want enforcement to be durable.

Related NHI Mgmt Group resources: AI Supply Chain Security and AI-BOM Guide · MCP Security Guide · NHI Authentication Guide · Agentic AI Security Guide

Key terms

• LLM Security: LLM Security is the set of controls used to protect large language models, their prompts, outputs, training data, and integrations from misuse or compromise. Defenders monitor for prompt injection, data leakage, unsafe tool use, and model abuse, and they restrict access, validate inputs, and log interactions to reduce unintended actions.
• Runtime Policy Enforcement: Runtime policy enforcement evaluates a request at the moment it is executed instead of relying only on preconfigured permissions. For AI agents, this allows decisions to reflect current context, target sensitivity, and behavioural signals rather than static assumptions.
• AI Artifact: An AI artifact is the governed package that contains the model, prompts, retrieval data, tools, configuration and related metadata. Treating the system as one artifact makes it possible to scan, approve and track the complete unit instead of letting changes drift across separate repositories and teams.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity and secrets management. It helps practitioners connect identity controls to the broader security programmes that now govern AI systems.