LLM security is a runtime governance problem, not a static scan

At a glance

What this is: This analysis argues that LLM security is a separate discipline because runtime behavior, tool access, and prompt injection create failure modes that traditional application testing misses.

Why it matters: For IAM, PAM, and broader security teams, the key issue is that LLMs can act as governed systems with access, so privilege, retrieval, and output controls must be managed like other high-risk identity surfaces.

By the numbers:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, and as quickly as 9 minutes in some cases.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, sharing sensitive data, or revealing access credentials.

👉 Read Equixly's analysis of LLM security risks, prompt injection, and agentic abuse

Context

LLM security is now a runtime governance problem because the model, the prompts, the tools it can call, and the data it can retrieve all form part of the attack surface. In practice, that means control failures can emerge after deployment even when the underlying code is unchanged, which is why traditional static testing is not enough for LLM applications.

The article’s core point is that prompt injection, sensitive data leakage, excessive agency, and unsafe output handling are not edge cases. They are structural risks that appear when LLMs are allowed to interpret instructions, access external content, and interact with downstream systems, making identity, privilege, and trust boundaries relevant to both AI and IAM programmes.

For identity and security teams, the important shift is that an LLM with tool access behaves like a governed workload with delegated authority, not like a simple application component. That makes access scope, auditability, and least privilege central concerns rather than secondary implementation details.

Key questions

Q: How should security teams handle prompt injection in production LLM applications?

A: Security teams should treat prompt injection as a runtime control issue, not a content-moderation problem. The practical response is to inspect prompts, retrieved content, and tool outputs for hostile instructions, then block or downgrade unsafe sessions before the model can act on them. The model should never be the only enforcement layer.

Q: Why do LLM agents create new identity and privilege risks?

A: LLM agents can call tools, access data, and trigger actions, so their permissions become part of the security boundary. If those permissions exceed the task, a single injected prompt can turn into unauthorized access or workflow abuse. Identity teams should therefore govern agent privilege with the same discipline used for other high-risk delegated access.

Q: How do security teams reduce the impact of unsafe LLM output handling?

A: They should block direct execution of model output until it has been validated, normalised, and checked against the intended action. This matters most when output can become SQL, shell commands, or code for downstream systems. The control is to treat the model as an untrusted source, not as an execution authority.

Q: Who is accountable when an LLM agent acts outside its intended scope?

A: Accountability sits with the organisation that grants the agent access, defines the workflow, and accepts the risk of delegated action. Regulators and auditors will expect evidence of ongoing testing, logging, and access control, especially where the agent can process personal data or influence regulated decisions.

Technical breakdown

Why prompt injection works in LLM applications

Prompt injection works because LLMs process system instructions and user-supplied content in the same token stream, without a built-in way to reliably distinguish trusted instructions from malicious ones. Direct injection uses attacker-crafted prompts, while indirect injection hides instructions inside retrieved documents, web pages, or emails. In retrieval-augmented generation, the model may trust content that the surrounding application did not intend to treat as instructions. Once the model follows hostile instructions, it can reveal sensitive data, change behaviour, or trigger tool calls that the developer never authorised.

Practical implication: test for both direct and indirect prompt injection wherever the model retrieves external content or can call tools.

Why excessive agency turns an LLM into a higher-risk identity surface

Excessive agency occurs when an LLM agent is given more access, capability, or autonomy than its task requires. The security issue is not just model output quality. It is delegated authority across APIs, databases, file systems, and external services. If prompt injection reaches an over-privileged agent, the model can chain actions across multiple systems with little friction. That makes the agent’s permission set, retrieval scope, and downstream execution rights the real control points, especially where the agent can perform actions without human review.

Practical implication: treat agent permissions like privileged access and constrain every tool call to the minimum task scope.

How output handling and supply chain weaknesses expand LLM attack paths

Improper output handling happens when an application trusts LLM output and passes it directly into another system, such as a shell, database, or JavaScript runtime. At that point, the model becomes an injection pathway rather than a decision aid. Supply chain risk adds another layer because models, datasets, plugins, and APIs can introduce compromise even if the core application is intact. In LLM systems, trust is distributed across many dependencies, so a weakness in one component can contaminate the whole workflow.

Practical implication: validate every LLM output before execution and vet external models, plugins, datasets, and APIs as production dependencies.

Threat narrative

Attacker objective: The attacker aims to make the LLM disclose protected context or perform actions that benefit the attacker while appearing to execute a normal user request.

1. Entry begins when an attacker places malicious instructions in user input or in external content that an LLM application retrieves and trusts.
2. Escalation occurs when the model follows those instructions and extends its behaviour into tool calls, data retrieval, or other downstream actions beyond the original task.
3. Impact appears as prompt leakage, unauthorized actions, sensitive data exposure, cost abuse, or corrupted business workflows.

NHI Mgmt Group analysis

LLM security is now a governance problem, not a model-only problem. The article shows that risk emerges when prompts, retrieval, and tool execution are stitched together into one runtime system. That means the control boundary sits around behaviour, access, and output handling, not around the model artifact alone. Practitioners should govern the full execution path, not just the model.

Excessive agency is the named design failure behind many LLM incidents. When an LLM can retrieve content, call tools, and trigger downstream actions with too much privilege, a single malicious prompt can cascade across systems. This is where identity governance intersects directly with agentic AI security, because the agent’s permissions matter as much as its model quality. Practitioners should treat agent privilege as a first-class risk.

Prompt injection exposes a trust-boundary problem that static application testing does not catch. Traditional security testing assumes code paths are stable, but LLM behaviour changes with context, input, and tool access. That creates a runtime trust boundary gap where valid-looking content can become instruction-level abuse. Practitioners should build governance around runtime evidence, not point-in-time validation.

System prompt leakage is more than information disclosure, it is control-plane exposure. Once attackers recover hidden instructions, they gain insight into business rules, safety constraints, and internal control logic. In security terms, this reduces uncertainty for the attacker and increases the chance of targeted exploitation. Practitioners should treat system prompts as sensitive operational assets, not implementation detail.

OWASP-style classification is useful only if it drives control decisions. The value of risk taxonomies lies in mapping them to testable controls for injection, output handling, privilege separation, and supply chain vetting. Without that mapping, LLM security stays descriptive instead of operational. Practitioners should anchor LLM governance to concrete control ownership and measurable test coverage.

What this signals

Runtime trust boundary gap: LLM programmes will increasingly be judged on what they can prove about prompts, retrieval, and tool execution, not on whether the model passed a one-time test. The organisations that build evidence trails now will be better positioned for audit, incident response, and policy enforcement later.

The immediate planning signal is that agent permissions and downstream execution paths need to sit inside the same governance conversation as model performance. As AI systems become more connected to business processes, identity and privilege control become the practical levers for containing blast radius.

Teams should expect more overlap between AI security and IAM in the next cycle of control design. If an LLM can retrieve data or trigger actions, then access reviews, audit logs, and approval boundaries must be explicit rather than implied.

For practitioners

• Build runtime adversarial testing into CI/CD Run prompt injection and tool-abuse tests continuously in production-like environments, not only during pre-deployment review, so hidden retrieval and tool-call failures are exercised before release. Use the same test cases to track regressions over time.
• Constrain agent permissions to task scope Map every external tool, API, file system, and database access path an LLM can reach, then reduce permissions to the minimum required for each workflow. Apply privileged access discipline to agents that can trigger downstream actions.
• Validate all downstream LLM output Prevent the model from passing raw output into execution layers such as shells, SQL interpreters, or browser automation without validation and sanitisation. Treat the model as an untrusted content generator until the output has been checked.
• Track every interaction and decision trail Log prompts, retrieved content, tool calls, model outputs, and remediation actions in a way that supports audit, incident response, and reproducibility. This is the evidence chain auditors need when LLM behaviour becomes contested.
• Vet the full LLM supply chain Review models, plugins, embeddings, datasets, and external APIs for provenance, access scope, and update risk. Supply chain control matters because compromise in any dependency can change the behaviour of the whole application.

Key takeaways

• LLM security fails at runtime because prompts, retrieval, and tool calls can be steered after deployment, even when code is unchanged.
• The evidence base is already strong: 31 of 36 tested applications were vulnerable to systematic prompt injection, showing the problem is operational, not theoretical.
• Security teams need to govern agent privilege, validate output before execution, and maintain interaction logs that support audit and incident response.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Excessive agency: A condition where an AI system is given more operational authority than its task requires. The risk is not just poor output. It is that mistakes, manipulation, or compromise can produce destructive actions at machine speed across the systems the agent can reach.
• System Prompt Leakage: System prompt leakage is the exposure of hidden prompt content to users or attackers. The real security problem is usually not the prompt itself, but the secrets, policy logic, and internal architecture details placed inside it. If those details are sensitive, they should live in code or secrets management instead.
• Improper Output Handling: A vulnerability where LLM-generated content is trusted before it reaches another system. The danger appears when that output is rendered, parsed, or executed without validation, encoding, or policy checks, allowing attackers to turn model responses into code, queries, or commands.

What's in the full article

Equixly's full blog covers the operational detail this post intentionally leaves for the source:

• The article’s step-by-step breakdown of the OWASP LLM attack categories and how each one maps to runtime failures.
• The testing approach used to probe production software with HOUYI and document reproducible evidence.
• The specific remediation guidance Equixly associates with prompt leakage, tool-call abuse, and unsafe output handling.
• The compliance context for EU AI Act, NIS2, and PCI DSS evidence expectations.

👉 The full Equixly post adds the OWASP mapping, test examples, and compliance context for implementation teams.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management. It gives security practitioners a practical way to connect delegated access, auditability, and identity control.