LLM security is a runtime governance problem, not a static scan

At a glance

What this is: Equixly shows that LLM security failures are primarily runtime failures, with prompt injection compromising most tested applications and exposing prompts, actions, and downstream systems.

Why it matters: IAM, NHI, and AI security teams need to treat LLMs as governed runtime actors because access scope, tool permissions, and output controls determine blast radius.

By the numbers:

• When researchers deployed HOUYI against 36 LLM-integrated applications, it succeeded on 31 of them.
• Wang et al. analyzed 529 vulnerabilities in 75 major LLM projects and found that improper resource control made up 45.7% of the issues.

Context

LLM security fails when organisations assume static code review can prove a system is safe after deployment. Large language model applications behave differently from conventional software because prompts, retrieved content, tool calls, and outputs all interact at runtime, which creates an attack surface that only appears under real use.

That matters for identity and access governance because an LLM integrated with tools is not just software, it is a decisioning layer that can inherit access, issue actions, and move data. The article shows that prompt injection, excessive agency, and output handling are governance problems as much as they are application security problems.

The article's starting position is typical of modern LLM deployments: the control gap is not unusual, it is structural.

Key questions

Q: What breaks when LLM security testing stops at static analysis?

A: Static testing misses the failure modes that emerge only when prompts, retrieval, and tools interact at runtime. That means prompt injection, prompt leakage, and unsafe tool execution can pass code review and still fail in production. Security teams need session-level evaluation because the real attack surface appears after the model assembles context and begins acting.

Q: Why do LLM agents create more privacy and compliance risk than standard chatbots?

A: LLM agents increase risk because they can read, store, and act on more data across more systems. Memory, multimodal inputs, and tool use expand the volume of sensitive information in play, while weak controls can expose personal data or credentials. That raises GDPR, audit, and deletion challenges, especially when decisions are made without clear human review or reliable data flow tracing.

Q: What are the signs that prompt injection defenses are failing in LLM applications?

A: Common signs include unexpected system instruction disclosure, unauthorized data access, abnormal tool use, and model outputs that ignore policy or safety boundaries. Teams should also watch for sensitive actions being attempted without human review and for audit logs that show conversations or prompts leading to unsafe downstream behavior. These signals indicate the model is accepting attacker-controlled instructions.

Q: How should security teams implement access controls for AI and LLM environments?

A: Security teams should treat AI and LLM access as a governed identity problem, not just an application setting. Start by centralising authentication with SSO and MFA, then apply RBAC and least privilege so users and services only reach the data and actions they need. Add regular access reviews and audits to remove stale permissions and reduce the blast radius of compromised accounts.

Technical breakdown

Why prompt injection defeats static assurance

Prompt injection works because many LLM systems treat user input, retrieved content, and system instructions as part of one conversational context. The model does not inherently know which text is trusted and which is hostile, so a malicious instruction can override intended behaviour if it enters through a channel the application accepts. Direct injection uses attacker-crafted prompts. Indirect injection hides the payload in external content that the application retrieves and processes during a legitimate task. In both cases, the vulnerability sits in the runtime trust boundary, not in a traditional code defect.

Practical implication: test the conversation path, retrieval path, and tool path as one control surface, not as separate assurance problems.

How excessive agency turns a model into a privilege carrier

Excessive agency occurs when an LLM agent can access APIs, databases, files, or external services beyond what the task requires. That becomes dangerous because the model can be manipulated into using those privileges in ways the developer never intended. In identity terms, the agent behaves like a non-human identity with borrowed authority. If access scope, authorization checks, and action approval are loose, a single injected prompt can cascade into data access or operational changes across connected systems. This is why agent design and privilege design are inseparable.

Practical implication: bind every tool call to a narrow task scope and verify that the agent cannot act outside its explicit authorization.

Why output handling is part of the attack path

Improper output handling turns LLM output into a transit point for downstream compromise. If a model produces SQL, shell commands, JSON, or HTML and the application executes or trusts that output without validation, the attack is no longer confined to text generation. The failure is architectural: the application treats model output as if it were already safe. That pattern is especially dangerous in systems that combine retrieval, agentic workflows, and external APIs because the model can be induced to generate payloads that downstream systems consume automatically.

Practical implication: enforce validation, allowlisting, and human review before model output reaches executable or privileged systems.

Threat narrative

Attacker objective: The objective is to hijack the model's runtime trust so the application reveals information, consumes resources, or executes actions outside developer intent.

1. Entry occurs when an attacker supplies crafted prompt content or malicious retrieved content to an LLM-integrated application.
2. Credential or authority abuse follows when the model interprets hostile instructions as legitimate and uses connected tools, prompts, or permissions accordingly.
3. Impact occurs when the application leaks internal prompts, discloses sensitive data, spends paid service credits, or performs unintended actions in downstream systems.

Breaches seen in the wild

• EchoLeak (Microsoft 365 Copilot) 2025: A crafted email could make Microsoft 365 Copilot leak data from its context with no click, a zero-click prompt injection fixed as CVE-2025-32711.
• Meta Muse agent hijack 2026: An undocumented Muse setting let local malware hijack Meta's personal AI agent, steal its authentication material and abuse user access.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Runtime governance is the real LLM security boundary: static review cannot prove how a model will behave once prompts, retrieval, and tool calls are live. The article's evidence shows that the exploitable condition is not the codebase alone, but the decision path created at inference time. Practitioners should treat LLM security as ongoing runtime governance, not a one-time assurance exercise.

LLMs become non-human privilege carriers the moment they can act on behalf of users: that is where identity governance becomes central. An agent with access to APIs, files, or paid services is effectively operating as a non-human identity with delegated authority, so privilege scope and authorization boundaries matter as much as model quality. Teams should govern agent identity and tool access together.

Prompt injection exposes a trust boundary problem, not just a content moderation problem: the model is not failing to detect bad language, it is failing to separate trusted instructions from untrusted inputs. That distinction is critical in RAG, browser, and tool-using systems where external content becomes executable context. The practical conclusion is that trust classification must move into the orchestration layer.

Output validation is now a core control for AI-enabled systems: once model output can drive code execution, workflow actions, or external API calls, the output channel becomes a privileged interface. This is the same governance mistake seen in other weakly mediated automation layers: unsafe trust in machine-generated instructions. Practitioners should map that failure mode to explicit control ownership.

Named concept, runtime trust collapse: this article shows how the boundary between trusted system instruction and untrusted external input collapses under prompt injection and agentic tooling. That collapse is what turns ordinary LLM use into security risk. The practitioner answer is to govern trust at runtime, not after the fact.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Red Teaming AI Agents for Identity Abuse

What this signals

Runtime trust collapse: the central governance issue in LLM security is the moment untrusted text becomes operational input. Once prompts, retrieved content, and tool calls share the same execution path, the model can be induced to act outside intended policy, so controls have to govern the whole chain rather than the prompt in isolation.

LLM programmes will keep failing if teams measure safety at the code layer while the real risk lives in orchestration. The operating model should shift toward continuous adversarial testing, strict tool authorization, and output validation tied to identity-aware policy enforcement.

For practitioners

• Run adversarial prompt tests in production-like conditions Exercise the application with direct and indirect prompt injection patterns against retrieval, tool use, and output handling before release.
• Restrict agent tool scope to the minimum task boundary Assign only the APIs, data sources, and actions required for the current task, and revoke everything else by default.
• Validate every model output before downstream use Treat generated text, JSON, SQL, and shell content as untrusted until it passes format checks, allowlists, and policy enforcement.
• Log prompts, tool calls, and output decisions as evidence Capture timestamped records that let reviewers reconstruct what the model saw, what it chose, and what it sent onward.
• Vet every third-party component in the LLM stack Track models, plugins, datasets, retrieval sources, and external APIs as governed dependencies with explicit ownership and review.

Key takeaways

• LLM security failures arise at runtime, where prompts, retrieval, tool use, and output handling interact in ways static review cannot fully predict.
• The article's evidence shows that prompt injection can compromise a large share of tested LLM-integrated applications, which makes runtime assurance a governance requirement, not a specialist exercise.
• Practitioners need to control agent authority, validate outputs, and keep full audit trails if they want LLM deployments to stay inside intended policy boundaries.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Excessive agency: A condition where an AI system is given more operational authority than its task requires. The risk is not just poor output. It is that mistakes, manipulation, or compromise can produce destructive actions at machine speed across the systems the agent can reach.
• Output Handling: Output handling is the control discipline for treating model responses as untrusted until they are validated, filtered, or constrained. In LLM environments, poor output handling turns text generation into an execution risk when downstream systems trust the response too early.
• Runtime Governance: Runtime governance is the set of controls that verify what a system or agent is actually doing after deployment. It combines monitoring, authorization checks, and access validation so teams can detect drift, misuse, or excessive privilege in motion rather than assuming build-time policy still holds.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and secrets management. It helps practitioners connect delegated access, runtime control, and identity risk across modern security programmes.