LLM security exposes the governance gap in AI and identity controls

At a glance

What this is: This is an LLM security overview that frames the main risks, control layers, and best practices for protecting models, applications, and agents.

Why it matters: It matters because security teams now have to govern AI systems as identity-bearing actors, not just software features, across NHI, autonomous, and human workflows.

By the numbers:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, and as quickly as 9 minutes in some cases.

👉 Read Lasso Security's guide to LLM security risks and best practices

Context

LLM security is the practice of protecting large language models and the applications built on them from prompt injection, data leakage, model poisoning, and excessive agency. In identity terms, the core problem is that AI systems increasingly touch secrets, data, and tools without being governed by the same controls that were designed for people or static workloads.

The article treats LLM security as a lifecycle problem, from training data and employee data handling to access control, monitoring, and response. That framing is useful because the risk is not just model compromise. It is identity misuse, policy bypass, and unsafe delegation across NHI, autonomous, and human-facing workflows.

Key questions

Q: How should security teams govern LLMs that can call tools and APIs?

A: Security teams should govern tool-enabled LLMs like delegated identities with explicit boundaries. Separate generation from execution, restrict each connector to the smallest possible scope, and require policy checks before any write, delete, or delegate action. If the model can reach sensitive systems, its permissions must be reviewed as carefully as any service account.

Q: Why do LLMs create more risk than ordinary automation workflows?

A: LLMs create more risk because their behaviour can change at runtime based on prompt content, retrieved context, or user input. That means the same workflow may produce different actions, exposures, or outputs under different conditions. Conventional automation is predictable. LLM-driven behaviour is more dynamic, so identity and data controls must account for shifting intent.

Q: What do teams get wrong about prompt injection?

A: Teams often treat prompt injection as a text-only problem, but it is really a trust-boundary problem. The attacker is trying to influence decisions, tool use, or data disclosure through the model interface. Defences have to include input filtering, output validation, and restrictions on what the model can do with retrieved or user-supplied content.

Q: Who is accountable when an AI system exposes data or takes an unwanted action?

A: Accountability should sit with the team that assigned the permissions, connected the data, and approved the workflow. In practice, that means product owners, security, and identity governance all share responsibility for the model’s delegated reach. If the permissions were excessive, the failure is governance, not just model behaviour.

Technical breakdown

Prompt injection and unsafe output handling

Prompt injection is a manipulation technique where attacker-controlled text steers an LLM into revealing data, ignoring policy, or taking unintended actions. Unsafe output handling makes this worse when downstream systems trust model output as if it were validated user input. In practice, the model becomes a control point that can be tricked into crossing trust boundaries. That is why LLM security has to include input sanitisation, output validation, and strict separation between generated text and executable instructions.

Practical implication: Treat model output as untrusted until it is checked by policy, parsing, and access controls.

Excessive agency in plugins, agents, and APIs

Excessive agency occurs when an LLM-connected system can call tools, access data, or perform actions beyond the task it was meant to complete. The security issue is not intelligence, but delegated power. Once tools are attached, the model can move from answering questions to making decisions that have real operational impact. This is where NHI-style controls become relevant, because the tool credentials, permissions, and boundaries determine the blast radius even when the model itself is the interface.

Practical implication: Limit tool scope, separate read and write paths, and bind every action to explicit policy.

Data security, training integrity, and model poisoning

LLM security also depends on the integrity of the data used to train, fine-tune, and operate the model. Model poisoning happens when malicious or corrupted data changes model behaviour, while data leakage occurs when sensitive information is exposed through training corpora, prompts, logs, or outputs. These are not abstract AI risks. They are governance failures around what enters the model, what it remembers, and what it is allowed to reveal. Security teams need to treat datasets, checkpoints, and logs as controlled assets.

Practical implication: Apply data classification, provenance checks, and access restrictions to training and runtime data paths.

Threat narrative

Attacker objective: The attacker seeks to turn the model into an access path for data exposure, unauthorised actions, or trusted workflow manipulation.

1. Entry occurs through prompt injection, sensitive data sharing, poisoned training content, or overly permissive tool integration that exposes model inputs or connected systems.
2. Escalation happens when the model is allowed to act on those inputs, use plugins or APIs, or leak information through outputs that downstream systems trust.
3. Impact follows as confidential data is disclosed, malicious content is generated, or the model performs unwanted actions that expand the attacker’s operational reach.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM security is becoming identity security by another name. Once a model can access data, call tools, or influence workflows, the question is no longer only whether the model is accurate. The question is who or what is authorised to act through it, and under what boundaries. That makes LLM security a control problem across NHI, IAM, and lifecycle governance, not just a model-risk problem. Practitioners should evaluate AI systems as governed identities with bounded privileges.

Excessive agency is the clearest governance failure in LLM deployments. The article correctly identifies that plugins, agents, and APIs become dangerous when they are given more authority than the task requires. This is where least privilege must be interpreted operationally, not abstractly: read, write, execute, and delegate are different powers. The implication is that security teams must map AI permissions to actual actions, not to product labels.

Training data security and runtime access control are the same control plane at different stages. If secrets, proprietary code, or sensitive records are allowed into model training, prompt logs, or connected tools without strong governance, the model becomes a persistence layer for exposure. That makes lifecycle discipline critical, from data ingestion through monitoring and offboarding of access paths. Practitioners should treat model supply chains like other identity supply chains.

Named concept: identity blast radius. In LLM environments, blast radius is defined by the combined reach of prompts, tool access, data permissions, and downstream trust in model output. The article shows why this matters: a model with narrow intent can still create broad impact if the surrounding access model is loose. The practical conclusion is to measure not just model behaviour, but the maximum damage reachable through its delegated identity.

From our research:

• Only 1.5 out of 10 organisations are highly confident in their ability to secure NHIs, compared to nearly 1 in 4 for securing human identities, according to The State of Non-Human Identity Security.
• Lack of credential rotation is cited as the top cause of NHI-related attacks by 45% of organisations, followed by inadequate monitoring and logging at 37% and over-privileged accounts at 37%.
• The practical next step is to pair LLM governance with lifecycle discipline, as outlined in NHI Lifecycle Management Guide, so tool access, ownership, and offboarding do not drift apart.

What this signals

Identity blast radius: LLM programmes should now be measured by the maximum downstream damage a model can cause through its connected tools, not by prompt quality alone. That means security and IAM teams need shared visibility into credentials, connectors, and approvals before models are allowed into production.

The governance pattern is shifting from static access review to continuous delegation review. If the model can retrieve, rewrite, or trigger actions using credentials embedded in workflows, the risk is no longer just data loss. It is uncontrolled authority spread across multiple identity layers.

Practitioners should expect policy pressure to converge around lifecycle controls for machine-facing access, especially where LLMs touch secrets, code, or customer data. The organisations that standardise that governance early will have clearer evidence of control when audits, incidents, or AI expansion forces the issue.

For practitioners

• Inventory every AI-connected identity path Map the credentials, service accounts, API keys, plugins, and data connectors that can be reached from each LLM workflow. Classify them by read, write, and delegation capability so you can spot where model access exceeds the task boundary.
• Separate prompts from execution privileges Ensure that a model can suggest actions without directly holding the permissions to perform them. Put approval gates or brokered services between generation and execution for any workflow that can change data, trigger systems, or expose records.
• Apply least privilege to tool-enabled agents Scope each agent or model integration to the minimum connector set, resource scope, and action set required. Revalidate permissions whenever the workflow changes, especially when the model is connected to customer data, code repositories, or operational systems.
• Harden training and prompt data pipelines Treat training sets, fine-tuning corpora, prompt stores, and logs as controlled inputs. Use provenance checks, access restrictions, and review processes so poisoned or sensitive data cannot silently shape model behaviour.
• Test for output misuse before production rollout Red team the model for prompt injection, unsafe output handling, and tool misuse using realistic attack paths. Verify what happens when the model is asked to reveal, transform, or relay sensitive information into downstream systems.

Key takeaways

• LLM security is an identity governance problem as much as a model-risk problem because tool access, data access, and delegated action all expand the attack surface.
• The biggest practical failure mode is excessive agency, where connected plugins and APIs let the model cross boundaries that should stay under explicit control.
• Security teams should govern model training, runtime access, and workflow delegation as one lifecycle so exposure does not move faster than review and offboarding.

Key terms

• Prompt Injection: Prompt injection is a technique that steers an LLM with attacker-controlled instructions hidden in input or context. The goal is to alter model behaviour, disclosure, or downstream actions by exploiting the model's trust in text, not by breaking the system directly.
• Excessive Agency: Excessive agency is the condition where an LLM-connected system has more power than the task requires. It becomes risky when the model can call tools, access data, or trigger actions that were never meant to be part of the workflow's normal authority.
• Model Poisoning: Model poisoning is the deliberate contamination of training or fine-tuning data so the model learns harmful, biased, or attacker-friendly behaviour. In practice, it affects integrity, can hide malicious patterns, and may persist across releases unless data provenance and review are tightly controlled.
• Identity Blast Radius: Identity blast radius is the maximum harm an identity can cause if it is misused, compromised, or over-scoped. For LLM systems, it includes the permissions behind prompts, the tools the model can call, and the trust placed in its outputs.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or governance in your organisation, it is worth exploring.

This post draws on content published by Lasso Security: Large Language Model (LLM) Security: Challenges & Best Practices. Read the original.