AI model security is becoming an identity governance problem

At a glance

What this is: This is a security and governance analysis of AI model protection, with the key finding that model risk now includes access control, misuse, and auditability across the AI lifecycle.

Why it matters: It matters because AI models sit inside business workflows, so IAM, PAM, and governance teams have to treat model access and runtime influence as identity problems, not just application security issues.

Context

AI model security is the set of controls that protect machine learning and generative AI systems from manipulation, leakage, and unauthorized use. The primary gap is that models are not just software artefacts, they are decision-making surfaces with inputs, outputs, APIs, training data, and runtime behaviour that all need governance.

For identity practitioners, the important shift is that model security now intersects with access management, privilege scope, and auditability across the full AI lifecycle. That includes human users, service identities, and AI systems that can act on models or data through APIs and orchestration layers.

Key questions

Q: How should security teams govern AI models that can call tools and access data?

A: Security teams should govern AI models as non-human identities with named owners, limited scope, short-lived credentials, and continuous authorization. The critical shift is to treat every tool call, data read, and update path as a privileged action that can be logged, revalidated, and revoked. Without that discipline, model risk becomes identity risk.

Q: Why is prompt injection a governance problem as well as a technical one?

A: Prompt injection is a governance problem because the harm depends on who gave the agent authority, what tools it can use, and how much data it can reach. A model can be influenced by language, but only governance determines whether that influence becomes a real security incident through overbroad permissions or weak oversight.

Q: What are the signs that model access is too broad?

A: Common signs include many unrelated applications calling the same model, weak logging, unclear ownership, and no documented reason for access. If teams cannot quickly explain who depends on the model and why, the exposure scope is already wider than governance can comfortably manage.

Q: What should organisations do when AI access and AI safety are owned by different teams?

A: They should define a shared governance model with separate control objectives. The identity team should own provisioning, authorization, auditability and revocation, while the AI assurance team should own adversarial testing, runtime monitoring and failure-mode analysis. Common reporting helps, but the controls should not collapse into one programme because they answer different risk questions.

Technical breakdown

Why AI model APIs become identity control points

AI models are commonly consumed through APIs, pipelines, and endpoints, which turns access control into a first-line security issue. If authentication, authorization, and logging are weak, the model becomes available to more users and systems than intended, including scripts, downstream apps, and internal automations. The same pattern applies when model outputs feed other workflows, because trust extends beyond the model itself into every caller and consumer. This is why AI model security is not only about inference quality or adversarial robustness. It is also about who can invoke the model, what they can pass to it, and what they can do with the output.

Practical implication: govern model endpoints like privileged services, with explicit access scope, monitoring, and revocation.

How prompt injection and data poisoning change the trust boundary

Prompt injection targets the instructions a generative model receives at runtime, while data poisoning corrupts the training or validation data that shapes model behaviour before deployment. Both attacks exploit the fact that AI systems learn from and respond to untrusted inputs across multiple stages, not just at login or API call time. That makes the trust boundary wider than a conventional application boundary. Security teams have to treat prompts, datasets, and model artefacts as governed inputs with provenance, validation, and integrity checks. If those controls are absent, attackers can influence model output without ever needing to compromise the model binary itself.

Practical implication: validate inputs, datasets, and artefacts separately instead of assuming one security control covers all three.

Why model theft is also a governance failure

Model theft and extraction attacks use repeated queries, exposed artefacts, or insecure deployment paths to recreate or copy model behaviour. That is not only an intellectual property problem. It also reveals that the organisation failed to control how the model was exposed, who could query it, and whether outputs could be harvested at scale. Rate limiting, logging, and encryption help, but the deeper issue is governance over model access and lifecycle boundaries. When a model can be queried as if it were public infrastructure, the organisation has effectively lost control over who can reproduce its capability.

Practical implication: inventory exposed models, cap query volume, and tie access to business justification and ownership.

Threat narrative

Attacker objective: The attacker wants to manipulate model behaviour, steal model capability, or use the model as a channel for data leakage and operational abuse.

1. Entry typically begins through legitimate model access, exposed APIs, or poisoned training data that the organisation accepted into the lifecycle without sufficient control.
2. The attacker then uses prompts, repeated queries, or malicious datasets to influence model behaviour, extract sensitive data, or reconstruct model capability.
3. Escalation occurs when the compromised model output is consumed by downstream workflows, spreading bad decisions, leakage, or automation errors into other systems.
4. The impact is distorted decisions, data exposure, model replication, or broader compromise of AI-enabled business processes.

Breaches seen in the wild

• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.
• LiteLLM PyPI package breach: LiteLLM PyPI supply chain attack, credentials stolen from users.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI model security is now an identity governance problem, not only a model integrity problem. Access, audit, and privilege boundaries increasingly determine whether a model can be trusted in production. If teams only test model behaviour and ignore who can reach it, they leave the governance layer undefended. Practitioners should treat model access as a governed identity surface.

Prompt injection and data poisoning expose a governance gap in the lifecycle, not just a runtime threat. Training data, prompts, and model artefacts all carry trust assumptions that can be violated before a model ever serves a user. That means the security programme must track provenance and authorisation across the full lifecycle. The practitioner conclusion is simple: if input lineage is unclear, model trust is already weakened.

Model APIs create identity blast radius when they are over-exposed. Every extra caller, integration, and automation increases the chance that model outputs are harvested, abused, or consumed out of context. This is where least privilege becomes operationally meaningful for AI systems. Security teams need to understand that the blast radius is defined as much by access scope as by model sophistication.

Runtime controls matter because AI systems collapse the old separation between application access and security decisioning. A model that can be queried, prompted, or indirectly steered is already participating in access governance. That is why monitoring, logging, and authorisation are not optional add-ons. In practice, AI model security has to be designed as part of the identity plane, not layered on top of it.

Identity governance for AI will increasingly look like lifecycle governance for non-human actors. The same questions that govern service accounts now apply to models, agents, and their surrounding APIs: who owns them, who can change them, and who can revoke them. The named concept here is model access blast radius: the scope of damage created when a model is reachable by too many identities and workflows. Practitioners should reduce that blast radius before AI usage spreads further.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Agentic AI Identity Maturity Model

What this signals

Model access blast radius: AI governance teams should treat every model endpoint as a privileged identity surface because overexposure increases both data leakage and output reuse. The programme signal is that access scope, not just model quality, now determines how far a failure can spread.

The rise of AI-related credential leaks shows why adjacent infrastructure matters as much as the model itself. According to the State of Secrets Sprawl 2026, AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers.

For practitioners

• Define model access as a privileged identity surface Classify model endpoints, orchestration hooks, and inference APIs as governed assets with named owners, explicit entitlements, and revocation procedures.
• Separate controls for prompts, data, and artefacts Apply different validation and approval checks to runtime prompts, training datasets, and model files so one trust decision does not cover all three.
• Inventory model exposure across internal and external callers Map every application, workflow, and automation that can reach a model, then remove access paths that are not tied to a documented business need.
• Instrument models for anomalous query and output patterns Log high-volume prompts, repeated extraction attempts, and unusual output sequences so abuse can be investigated before model behaviour is widely reused.
• Tie model lifecycle changes to approval and ownership Require formal review before models, prompts, or API integrations are changed so governance keeps pace with deployment and reuse.

Key takeaways

• AI model security now spans access, influence, and auditability, not just output correctness or adversarial robustness.
• The most consequential failures arise when untrusted prompts, poisoned data, or exposed APIs reshape model behaviour before governance can intervene.
• Identity teams should reduce model exposure, separate lifecycle controls, and treat AI endpoints as privileged services with explicit ownership.

Key terms

• AI Model Security: AI model security is the practice of protecting machine learning and generative AI models from theft, tampering, misuse, and unsafe behavior. It covers the model lifecycle, including training data, weights, prompts, inference endpoints, and update paths, with controls for access, integrity, monitoring, and resistance to attacks such as poisoning, extraction, and prompt injection.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Data Poisoning: The deliberate or accidental contamination of a data source that influences system behaviour. In AI environments, poisoned content can alter retrieval results, generated answers, or downstream decisions, which makes write access, change monitoring, and source integrity part of the identity control problem.
• Model Extraction: Model extraction is the theft of model behaviour or parameters through repeated interactions, often via an exposed API. The attacker may not need direct file access if the interface leaks enough information through outputs, rate limits, or weak request controls.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.