Why traditional IAM fails for generative AI security

At a glance

What this is: This is an independent analysis of why generative AI breaks traditional security models and what that means for securing LLM deployments.

Why it matters: It matters because IAM, NHI, and security teams now have to govern AI workloads, service accounts, and data pipelines that behave differently from conventional applications.

By the numbers:

• 82% of executives say secure and trustworthy AI is essential to their business, but only 24% of current generative AI projects have a security component built in.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
• 72% of organisations have experienced or suspect they have experienced a breach of non-human identities, with 46% confirmed and 26% suspected.

👉 Read Orca Security's analysis of generative AI security and LLM governance

Context

Generative AI security is an identity and data governance problem as much as it is an application security problem. Large language models accept free-form inputs, pull context from shifting sources, and act through service accounts, API keys, and third-party integrations that existing controls often do not fully describe.

That mismatch is why traditional perimeter, signature, and quarterly review models fall short. For security teams, the issue is not just model misuse. It is that AI workloads expand the effective identity surface while making provenance, access scope, and runtime behaviour harder to pin down.

IBM Institute for Business Value’s 2024 research reflects the operational gap: leaders want trustworthy AI, but most projects still begin without security built in. That is an unusually common starting position for a technology stack that can ingest sensitive data, reach external tools, and generate actions at machine speed.

Key questions

Q: How should security teams govern generative AI workloads without breaking existing IAM models?

A: Start by treating AI workloads as a distinct identity surface with their own service accounts, API keys, data flows, and approved tools. Then apply least privilege, zero trust, and data classification together instead of as separate programmes. The goal is not to force legacy controls onto AI, but to govern model access, retrieval, and downstream action as one risk domain.

Q: Why do generative AI systems create more security risk than traditional applications?

A: Because their behaviour is probabilistic, context-dependent, and often connected to external data sources and tools. Traditional applications are easier to validate because inputs and outputs are more predictable. In generative AI, the same user request can lead to different outputs and even different downstream actions, which makes static controls less reliable.

Q: What breaks when employees use public AI tools with sensitive company data?

A: The organisation loses control over where the data is stored, how long it persists, and whether it may be reused in ways the business cannot govern. Once sensitive material enters an unmanaged model, it may be logged, retained, or exposed through later interactions. That is why Shadow AI is both a data risk and an identity governance failure.

Q: Who is accountable when an AI model leaks data or triggers an unauthorised action?

A: Accountability should sit with the business owner of the AI use case, the team managing the underlying identities, and the security function that approved the control boundary. If the model can access data or tools, ownership cannot be vague. The programme must assign responsibility for discovery, access review, monitoring, and response before deployment.

Technical breakdown

Why LLMs break deterministic security models

Traditional applications behave predictably because code paths, inputs, and outputs are largely stable. LLMs are different: the same prompt can produce different outputs, and the model may pull context from vector databases, APIs, or retrieved documents at inference time. That means the security boundary is no longer just the application container or web request. It extends into prompt context, retrieval data, model weights, and downstream tools. Controls built for static systems struggle when the system’s behaviour changes with every call.

Practical implication: map AI-specific data flows and access paths before you assume conventional app controls are enough.

Prompt injection and downstream action abuse

Prompt injection works because models treat retrieved text and user input as part of the same reasoning context unless explicit guardrails intervene. Direct injection targets the prompt itself, while indirect injection hides malicious instructions in documents, tickets, or web content that the model later ingests. The real risk appears when the model is connected to tools such as code runners, CRM systems, or ticketing platforms. At that point, a language manipulation attack can become an unauthorised action in another system.

Practical implication: separate model reasoning from tool execution with input filtering, output checks, and strict tool authorization.

Shadow AI, training data poisoning, and model inversion

Shadow AI creates invisible exposure when employees use unsanctioned models with sensitive data. Training data poisoning is different but related: an attacker with overly broad credentials can taint datasets before training or fine-tuning, shaping the model’s future behaviour. Once deployed, the model may also leak memorised content or enable model inversion, where attackers infer sensitive input features from outputs. These are governance failures because identity, data classification, and pipeline controls were missing before the model ever learned anything.

Practical implication: govern AI data ingress, pipeline access, and sanctioned tooling as one control plane rather than separate security tasks.

Threat narrative

Attacker objective: The attacker wants either sensitive data exposure, corrupted model behaviour, or unauthorised downstream actions through the AI stack.

1. Entry occurs when an attacker reaches AI-connected data or a model interface through over-permissive cloud credentials, exposed prompts, or unsanctioned public AI tools.
2. Credential abuse or prompt misuse then lets the attacker inject instructions, taint training data, or drive the model toward unsafe retrieval and tool actions.
3. Impact follows when the model reveals sensitive data, performs unintended downstream actions, or embeds attacker influence into future outputs and training runs.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• DeepSeek breach — DeepSeek breach exposed 1M+ log lines and sensitive secret keys.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Traditional IAM was designed for predictable workloads, not probabilistic model behaviour. The article’s core point is that LLMs change inputs, outputs, and data dependencies at runtime, which makes static entitlement thinking incomplete. Least privilege still matters, but it no longer describes the full security problem when the system can retrieve, infer, and act in ways that were not known at provisioning time. Practitioners need to treat AI as a distinct identity and data governance surface, not as a normal application with a new interface.

Shadow AI is a governance failure before it becomes a security incident. If employees can move sensitive material into public models without IT visibility, the organisation has already lost control of data residency, retention, and downstream exposure. That is a policy, discovery, and lifecycle problem, not just a detection problem. The implication is that sanctioned usage, discovery, and data handling rules must be defined before AI adoption becomes habitual.

Ephemeral prompt trust debt: the model can consume and expose information faster than legacy review cycles can govern it. This is the named concept that best captures the article’s operational reality. The security stack may observe the environment, but it often cannot judge the semantic trustworthiness of model inputs and outputs in time to prevent leakage or misuse. Practitioners should treat prompt context as a high-risk data plane, not a benign user interface.

AI security posture management has become a control layer, not a point solution. The article correctly separates visibility, data protection, guardrails, and runtime monitoring because no single control stops prompt injection, poisoned data, shadow usage, and output leakage at once. That is an important discipline shift for identity and cloud teams, who often assume one policy domain can absorb another. The practical conclusion is that AI governance has to be coordinated across IAM, NHI, DSPM, and operations.

OWASP and NIST frameworks now need to be read through an identity lens for AI workloads. The article’s framework mapping is directionally right, but the underlying lesson is broader: once AI systems touch credentials, APIs, and sensitive content, they belong in the same governance conversation as workload identity and zero trust. That collapses the old separation between application security and identity security. Practitioners should align AI controls with identity governance instead of treating them as a parallel programme.

From our research:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials, according to AI Agents: The New Attack Surface report.
• 48% of companies say they cannot track and audit the data their AI agents access, creating a complete blind spot for compliance and breach investigation.
• That is why OWASP Agentic AI Top 10 is useful as a companion lens when AI behaviour starts crossing into runtime decision and tool-use risk.

What this signals

Ephemeral prompt trust debt: AI programmes accumulate risk when teams assume prompt context is short-lived enough to review after the fact. In practice, model access, retrieval, and downstream tool use can all occur before a human review cycle ever starts, which means governance has to move upstream into discovery, classification, and entitlement design.

With 48% of companies unable to track and audit the data their AI agents access, according to our AI Agents: The New Attack Surface report, the operational question is no longer whether AI needs controls. It is whether the programme can still prove where the data went, which identity used it, and what action the system took.

For teams aligning AI governance with identity architecture, the next step is to connect AI-specific discovery to broader workload identity and zero trust work. The practical pattern is to keep AI posture, secrets, and access review in the same operating rhythm rather than allowing each to drift into a separate queue.

For practitioners

• Inventory every AI workload and shadow AI path Create a live inventory of model endpoints, RAG services, vector stores, API keys, and unsanctioned AI tools. Tie each to an accountable owner and a business purpose so hidden AI use cannot sit outside governance.
• Scope service accounts to the narrowest AI function Review the service accounts and machine identities used by training, inference, and retrieval services. Remove broad storage and network permissions, then separate data engineering, model training, and inference access into distinct roles.
• Classify training inputs before they reach the pipeline Scan data lakes, document stores, and vector databases for sensitive content before fine-tuning or retrieval indexing begins. Quarantine or mask regulated data so the model never learns more than it should.
• Add prompt and output controls around tool-connected models Filter prompts for injection patterns, inspect outputs for sensitive data, and restrict which downstream tools the model can call. Keep tool execution separate from model reasoning so text manipulation does not become unauthorised action.
• Monitor runtime drift in model and access behaviour Baseline token usage, response patterns, retrieval patterns, and API call frequency, then alert on changes that may indicate poisoning, jailbreak attempts, or extraction activity. Use the findings to trigger policy review rather than relying on periodic audits.

Key takeaways

• Generative AI breaks deterministic security assumptions because the same identity, prompt, and data path can produce different behaviours at runtime.
• The evidence points to a governance gap, not just a tooling gap: most organisations still deploy AI before security, visibility, and accountability are fully in place.
• Security teams should govern AI as an identity, data, and runtime control problem, with discovery, least privilege, guardrails, and monitoring working together.

Key terms

• Shadow AI: Unapproved AI tools, model deployments, or integrations that operate outside formal security oversight. In practice, shadow AI creates blind spots for data handling, access review, and incident response because the organisation cannot reliably see what was used, by whom, or with which information.
• Prompt Injection: A technique that places malicious instructions into a prompt or retrieved context so the model follows attacker intent instead of the intended system rules. In enterprise environments, the risk grows when the model can call tools or reach downstream services, turning text manipulation into unauthorised action.
• Data Security Posture Management: A control discipline for finding, classifying, and remediating sensitive data before it is used by AI systems. For generative AI, DSPM matters because training and retrieval data shape what the model can reveal later, making upstream data hygiene part of model governance.
• AI Security Posture Management: A programme for discovering AI assets, measuring their exposure, and applying controls across model, data, identity, and runtime layers. It is broader than application security because it has to cover model endpoints, training pipelines, retrieval sources, and shadow deployments as one governed surface.

Deepen your knowledge

NHI governance, agentic AI identity, machine identity security, IAM, and identity lifecycle management are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or governance in your organisation, it is worth exploring.

This post draws on content published by Orca Security: Why Generative AI Breaks Traditional Security Models. Read the original.