AI model threats expose the identity gap in machine security

At a glance

What this is: This is a CyberArk analysis of five threat patterns against AI models, centered on the claim that machine identity controls are the foundation for securing AI systems.

Why it matters: For IAM and NHI practitioners, it frames AI models, agents, and connected systems as identity-governed assets rather than standalone software.

👉 Read CyberArk's analysis of the emerging AI threatscape for model security

Context

AI model security fails when organisations treat models as isolated applications instead of identity-bearing machines. Once AI systems can read training data, call tools, or influence downstream workflows, the governance problem becomes one of machine identity, access scope, and trust validation across the full execution chain. That makes this a direct NHI governance issue, not only a model risk issue.

CyberArk's article uses several attack patterns to show how adversaries can target the model itself or the data and controls around it. The key takeaway for practitioners is that AI security cannot depend on static trust assumptions, because a model that is authenticated once can still be poisoned, hijacked, or pushed beyond its intended guardrails later.

Key questions

Q: How should security teams govern AI models that can call tools and access data?

A: Security teams should govern AI models as non-human identities with named owners, limited scope, short-lived credentials, and continuous authorization. The critical shift is to treat every tool call, data read, and update path as a privileged action that can be logged, revalidated, and revoked. Without that discipline, model risk becomes identity risk.

Q: Why do AI models create new IAM and NHI risks?

A: AI models create new IAM and NHI risks because they can act, decide, and chain actions through connected systems without a human in the loop. That expands the attack surface from login events to runtime behaviour, data provenance, and tool access. Traditional IAM alone cannot govern autonomous execution authority well enough.

Q: What is the difference between model security and machine identity security?

A: Model security focuses on the integrity and behaviour of the AI system itself, while machine identity security governs the credentials, certificates, and trust relationships that let the system act. In practice, both are needed, but identity security is what limits blast radius when a model is compromised or misused.

Q: When should organisations apply zero standing privilege to AI systems?

A: Organisations should apply zero standing privilege whenever an AI system can access sensitive data, trigger operational actions, or invoke external tools. If the system does not need persistent access, it should not have it. Ephemeral access reduces the damage from prompt injection, credential theft, and runaway automation.

Technical breakdown

Why machine identity is the control plane for AI models

The article treats AI systems as machines that must be identified, verified, and trusted continuously. That is the right frame. Once a model is connected to data pipelines, APIs, and downstream tools, the security boundary shifts from the model binary to the identities and certificates that allow it to act. If those credentials are weakly governed, an attacker does not need to 'break' the model in the abstract. They only need to compromise the trust relationships around it. In NHI terms, the model becomes another non-human identity with execution authority, and that authority needs policy, expiry, and auditability.

Practical implication: Inventory every AI-connected identity, then bind it to explicit authorization policy, rotation, and monitoring.

How poisoning and tampering change the trust model

Poisoning attacks alter training or ingestion data so the model learns the wrong behaviour, while tampering changes a deployed system so it behaves incorrectly at runtime. Both are identity problems as much as data problems, because the attacker is exploiting trusted data flows and privileged write paths. The security failure is not just input corruption. It is the absence of strong provenance, validation, and separation between who can feed the system and who can change its behaviour. For practitioners, this means securing the identities that publish, update, approve, and retrain model inputs is as important as protecting the model endpoint itself.

Practical implication: Apply provenance checks and least privilege to every data and retraining path feeding AI systems.

Breakout and fugitive scenarios show what happens when guardrails fail

Breakout attacks aim to make a model ignore its intended restrictions, while fugitive scenarios describe a system escaping confinement and acting with broader access than intended. These are different failure modes, but both depend on excessive trust and poor containment. If a model can reach sensitive data, trigger actions, or chain through tools without continuous revalidation, the blast radius expands quickly. That is why static approval is insufficient. AI systems need runtime checks, scope boundaries, and kill-switch logic that can revoke access when behaviour drifts or policy is violated.

Practical implication: Design AI access as ephemeral and revocable, with runtime controls that can stop abuse before lateral impact spreads.

Threat narrative

Attacker objective: The attacker wants to subvert trusted AI behaviour, extract sensitive model or training data, and use the model's access to amplify harm across connected systems.

1. Entry occurs when attackers compromise the model's trusted data pipeline, attached credentials, or interaction path rather than attacking the model in isolation.
2. Escalation follows when poisoned inputs, guardrail bypasses, or over-broad tool access let the model perform actions beyond its intended policy scope.
3. Impact appears when the model exposes private training material, manipulates downstream decisions, or operates outside containment with broader system access.

Breaches seen in the wild

• Cisco DevHub NHI breach — IntelBroker exploited exposed Cisco credentials, API tokens and keys in DevHub.
• DeepSeek breach — DeepSeek breach exposed 1M+ log lines and sensitive secret keys.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI model security is now an identity governance problem, not a model-only problem. The article correctly shifts the conversation from algorithmic risk to machine trust. Once models interact with data, tools, and other systems, their identities and permissions become the real control surface. Practitioners should govern AI systems as non-human identities with explicit scope, lifecycle, and revocation requirements.

Poisoning attacks make provenance part of the access model. If an attacker can influence training data, instruction sources, or update paths, the model's output integrity is already compromised. That means identity controls must extend to writers, pipelines, retraining jobs, and approval workflows. The governance lesson is simple: trust in AI starts with who can feed it, not only who can query it.

Breakout risk creates a new form of identity blast radius. A model that can escape its intended guardrails or call tools beyond scope can become a high-speed privilege escalator. The field needs a sharper concept here: identity blast radius: the amount of downstream access and action a model can reach once its trust boundary fails. Practitioners should reduce that radius through segmentation, short-lived credentials, and continuous authorization.

Static approval is insufficient for autonomous systems. The article's focus on continuous authentication is directionally correct because AI behaviour changes with context, prompts, and data. IAM and PAM controls designed for human sessions do not map cleanly to machine decision loops. Organisations need runtime governance that can re-evaluate access when model behaviour changes, not only when credentials expire.

AI security programmes will converge with NHI governance faster than most teams expect. The same controls that matter for service accounts, API keys, and workload identities now matter for AI systems that can invoke tools and alter data. That convergence is not cosmetic. Teams that already manage NHI lifecycle discipline will move faster, while those treating AI as a separate security island will accumulate avoidable trust debt.

From our research:

• The average organisation believes more than 1 in 5 of their non-human identities are insufficiently secured, according to The 2024 ESG Report: Managing Non-Human Identities.
• 72% of organisations have experienced or suspect they have experienced a breach of non-human identities, according to the same report.
• For a broader view of real-world compromise patterns, see The 52 NHI breaches Report and compare the control failures with your own environment.

What this signals

AI security programmes will be judged by how quickly they absorb NHI discipline. As AI systems gain tool access, the practical question is no longer whether they are intelligent enough to help, but whether their identities are governed tightly enough to contain misuse. Teams should expect the boundary between model security and identity governance to keep shrinking, especially in environments that already struggle with service account sprawl and weak lifecycle controls. That is why the NHI control problem now reaches into AI operations.

Identity blast radius becomes the operational metric that matters most. If an AI system can reach multiple services, the organisation has created a concentrated trust failure point. Practitioners should measure how far a compromised model, token, or agent can move before revocation takes effect, then use that measurement to drive segmentation and least privilege. Zero Trust Architecture only works here if every machine action is continuously re-authorised.

With machine identities already showing up as insufficiently secured in more than 1 in 5 cases according to The 2024 ESG Report: Managing Non-Human Identities, AI expansion adds another layer of unmanaged trust. The programme response is to converge AI governance, NHI lifecycle management, and privileged access review into one operating model.

For practitioners

• Define AI models as governed non-human identities Create an inventory of every model, agent, and supporting workload identity. Attach owners, purpose, data access scope, and revocation criteria so the AI estate is managed like any other privileged NHI population.
• Constrain training and ingestion paths Restrict who can write to training data, retrieval sources, and model update pipelines. Add provenance checks, approval gates, and logging for every trusted input path that can influence model behaviour.
• Use ephemeral credentials for model tool access Issue short-lived tokens for API calls, database queries, and orchestration actions made by AI systems. Remove standing credentials from model workflows wherever possible and require revalidation for each sensitive action.
• Segment model privileges by task and environment Separate development, testing, and production access, and keep models from inheriting broad network or data permissions. Limit each model to the smallest tool set needed for its current job.
• Build runtime kill-switches and behavioural alerts Monitor for jailbreak patterns, unexpected data access, and abnormal tool use. When a model deviates from approved behaviour, revoke its credentials and stop downstream execution automatically.

Key takeaways

• AI model risk is inseparable from machine identity governance once models can read data, call tools, or influence workflows.
• Poisoning, tampering, breakout, and fugitive scenarios all expand blast radius when trust paths are too broad or too static.
• Security teams should treat AI systems as revocable non-human identities with provenance controls, short-lived access, and runtime containment.

Key terms

• Machine Identity: A machine identity is the credentialed trust record a non-human system uses to authenticate and act. In AI environments, that includes certificates, tokens, and service identities tied to models, pipelines, and agents. It is the control point that determines what the system can reach and when that access should end.
• Model Poisoning: Model poisoning is the deliberate corruption of training or ingestion data so an AI system learns or behaves incorrectly. The attack targets the trust chain before or during model training, which means the resulting failure can persist into production outputs, decisions, and automated actions.
• Breakout Attack: A breakout attack is an attempt to push an AI system past its intended safety boundaries so it reveals restricted data or performs disallowed actions. The risk is not only content abuse. It is the possibility that the model will operate outside its policy scope and extend access into connected systems.
• Identity Blast Radius: Identity blast radius is the amount of damage a compromised identity can cause before it is revoked or contained. For AI and NHI programmes, it describes how far a token, certificate, workload, or agent can move through systems, data, and tools once trust fails.

Deepen your knowledge

AI model threats and machine identity governance are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are trying to secure autonomous systems that can call tools and access data, this is a practical place to start.

This post draws on content published by CyberArk: The Emerging AI Threatscape: 5 New Risks to AI Models. Read the original.