AI risks are becoming identity risks in generative systems

At a glance

What this is: This is Lakera’s analysis of the major risk categories in AI and LLM deployments, with the key finding that runtime abuse, data leakage, and poisoned inputs require controls that go beyond traditional model testing.

Why it matters: It matters to IAM, PAM, NHI, and AI security teams because AI systems increasingly act as access-bearing software, which means credential scope, trust boundaries, and authorization decisions now shape AI risk.

👉 Read Lakera's analysis of AI risk categories and LLM failure modes

Context

AI risk is no longer limited to model accuracy or content safety. Once a model or application can retrieve data, call tools, or influence decisions, the governance problem shifts to access, privilege, and trust boundaries, which is where identity security becomes relevant.

That is why AI security cannot be treated as a separate lane from IAM and NHI governance. The same questions that apply to service accounts, API keys, and delegated access now apply to AI systems that can reach sensitive data, external services, or downstream workflows.

Key questions

Q: How should security teams govern AI models that can call tools and access data?

A: Security teams should govern AI models as non-human identities with named owners, limited scope, short-lived credentials, and continuous authorization. The critical shift is to treat every tool call, data read, and update path as a privileged action that can be logged, revalidated, and revoked. Without that discipline, model risk becomes identity risk.

Q: Why do conversational AI systems create new identity and access risks?

A: Because they can combine data retrieval, decision-making, and execution in a single interaction. That collapses the gap between information access and business action, which traditional IAM and security tools were not built to manage. The result is higher exposure when the system can modify records or disclose sensitive guest data.

Q: What breaks when prompt injection reaches a tool-using AI agent?

A: What breaks is the assumption that the model's output is low impact. Once the agent can call tools, a malicious instruction can become a database query, a file write, an email, or a deployment action. Without policy checks and approval gates, the agent's legitimate permissions become the attacker's path to impact.

Q: How do teams know whether AI governance is actually working?

A: Look for evidence that every AI interaction can be traced end to end, from identity and intent to output and enforcement. If auditors can ask for a transaction and receive a complete record in hours, not weeks, the programme is producing usable control evidence rather than just documentation.

Technical breakdown

Prompt injection and runtime control failure

Prompt injection works by manipulating the instructions an LLM receives at runtime so that untrusted content can override the system’s intended behaviour. Indirect prompt injection is especially risky because the malicious instruction can sit inside retrieved documents, webpages, or tool output, where the model treats it as context rather than attacker-controlled input. The failure is not just model misunderstanding. It is a boundary problem between data, instructions, and action. When the model can call tools or trigger workflows, prompt injection becomes an access-control issue as much as a content-safety issue.

Practical implication: isolate untrusted context from action-bearing prompts and treat tool invocation as a separately authorised decision.

Training data poisoning and model supply chain risk

Training data poisoning occurs when malicious, biased, or corrupted data is introduced into the training or fine-tuning pipeline so that the model learns the wrong associations or behaviour. The same risk appears in modern GenAI supply chains when embeddings, retrieval corpora, or fine-tuning datasets are assembled from mixed provenance sources. This is not only a data-quality concern. It is a governance problem because the model’s behaviour depends on data lineage, source trust, and pipeline integrity. A poisoned dataset can quietly change outputs long after deployment.

Practical implication: apply provenance checks, dataset approvals, and integrity controls to every training and retrieval source.

Why AI systems need identity-aware authorization

An AI application that can use tools is not just a model wrapped in a UI. It becomes a software actor that may hold API keys, delegated tokens, or service credentials, which means its access path must be governed like any other privileged workload. If those credentials are broad, persistent, or shared across environments, the AI system inherits the same blast-radius problems that affect poorly managed NHIs. This is where identity governance, secrets management, and least privilege intersect directly with AI safety.

Practical implication: give AI systems workload identities with narrowly scoped, short-lived privileges rather than static shared secrets.

Threat narrative

Attacker objective: The attacker wants to turn the AI system’s own trust in context, tools, and credentials into unauthorised access or harmful output at scale.

1. Entry begins when an attacker slips malicious instructions into retrieved content, training data, or user prompts that the model will later consume.
2. Escalation occurs when the model follows those instructions and invokes tools, exposes data, or reuses credentials beyond the intended scope.
3. Impact follows when the AI system leaks sensitive information, performs unauthorised actions, or amplifies poisoned outputs into downstream workflows.

NHI Mgmt Group analysis

AI governance debt is now an identity problem. Organisations often frame GenAI risk as a model safety issue, but the stronger control failure is that AI systems are being granted access without lifecycle governance. Once an AI application can query data, trigger actions, or call services, the programme needs the same discipline used for privileged accounts and non-human identities. The practical conclusion is that AI risk registers must include identity, privilege, and credential ownership.

Prompt injection becomes more dangerous when the model can act. A harmless-looking content manipulation becomes a security event when the model is allowed to issue tool calls, retrieve internal data, or approve downstream steps. That means the security boundary is not the prompt alone. It is the combination of prompt, retrieval, policy, and execution rights. Practitioners should treat model action paths as authorised workflows, not as incidental features.

Unscoped AI access expands blast radius faster than traditional application sprawl. AI systems can be connected to many sources quickly, which creates a hidden accumulation of permissions, secrets, and delegated trust. This is the same failure pattern seen in unmanaged NHIs, only with faster iteration and less visibility. The named concept here is AI privilege drift, where model-connected services accumulate access faster than governance can review it. Teams should make this drift visible before it becomes normalised.

Data provenance is now part of security governance, not just model quality. Poisoned or low-trust training data can change model behaviour in ways that are hard to detect after deployment, especially in retrieval-augmented systems. That raises the bar for approval, logging, and source validation across the AI supply chain. The practical conclusion is that model governance must be joined to data governance and identity governance in one control model.

The AI control stack should mirror zero trust principles without pretending the model is trusted. Zero trust is not about trusting the model less in the abstract. It is about constraining what the system can reach, what context it can consume, and what actions it can take. That puts policy enforcement, short-lived credentials, and runtime monitoring at the centre of AI security. Practitioners should assume every connected AI workflow needs explicit authorisation.

What this signals

AI privilege drift: AI systems accumulate access quickly because every new connector, tool, and retrieval source widens the trust boundary. The programme risk is not a single misconfiguration but a steady expansion of what the model can reach, which makes identity governance and lifecycle control part of AI security architecture.

Security teams should expect AI risk reviews to converge with IAM, PAM, and NHI governance reviews. When a model can use service credentials, the practical question is no longer whether the output looks safe, but whether the access behind it is scoped, logged, and revocable. That is a control problem, not a content problem.

The most resilient programmes will align AI controls with NIST Cybersecurity Framework 2.0 and identity-guided authorisation, then treat every external action as a monitored event. That approach reduces the chance that a prompt becomes an unauthorised workflow step.

For practitioners

• Separate context from execution Route prompts, retrieval results, and tool permissions through different control layers so that untrusted text cannot directly trigger privileged actions. This reduces the chance that injected instructions become operational commands.
• Issue workload identities to AI systems Assign each AI service its own workload identity, then scope tokens and API keys to a single application, environment, and purpose. Avoid shared secrets across agents, test systems, and production workflows.
• Approve data sources before training or retrieval Create an intake process for training sets, embeddings, and retrieval corpora that validates provenance, ownership, and integrity before the data reaches the model pipeline.
• Log and review model tool calls Record every external action the AI system takes, including API requests, delegated authorisations, and writes to downstream systems, so that teams can trace abuse and anomalous behaviour.

Key takeaways

• AI risk becomes materially harder to govern once models can access tools, data, and credentials.
• The biggest failure mode is not only unsafe output, but the collapse of boundaries between untrusted input and privileged action.
• Identity-aware authorization, data provenance, and short-lived workload credentials are now core controls for AI security.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Training Data Poisoning: Training data poisoning is an attack that corrupts the data an AI model learns from so it produces attacker-influenced results later. The corruption may be inserted during training or at runtime, and the model can appear normal while embedding a hidden failure path in its outputs.
• Privilege Drift: Privilege drift is the gradual gap between the permissions an identity was meant to have and the permissions it actually retains. In AI agent environments, drift grows quickly because roles are reused, tasks change, and lifecycle reviews often lag behind deployment velocity.
• Workload Identity: The identity assigned to a software workload — such as a containerised application, serverless function, or microservice — enabling it to authenticate to other services without storing static credentials.

What's in the full article

Lakera's full article covers the operational detail this post intentionally leaves for the source:

• Specific examples of prompt injection, jailbreaking, and in-context learning failures in live GenAI workflows
• A deeper breakdown of training data poisoning, model misuse, and content moderation failure modes
• Practical guidance on monitoring LLM behaviour when systems begin to act on retrieved content or tool output
• Implementation-oriented context on how AI security controls differ from traditional application security

👉 Lakera's full article covers the practical examples, threat categories, and AI security guidance in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security practitioners apply identity discipline to AI-connected systems and broader access governance.