AI security incidents expose the human and supply chain risk gap

At a glance

What this is: Proofpoint argues that the Anthropic leak and Mercor supply chain attack show AI security risk is being introduced through human error, insecure integrations, and compromised dependencies rather than through model weakness alone.

Why it matters: For IAM, NHI, and AI security teams, this matters because AI workflows inherit trust from the people, services, and integrations that connect them, so exposure control now depends on identity, data, and supply chain governance together.

By the numbers:

• 82% of breaches involve human factors such as error, misdelivery, or misuse.
• 17 minutes and as quickly as 9 minutes
• 72% of organisations have experienced or suspect they have experienced a breach of non-human identities.

👉 Read Proofpoint's analysis of the Anthropic leak and Mercor AI supply chain attack

Context

AI security incidents are increasingly caused by ordinary operational failures that become more damaging once AI systems are embedded into business workflows. The primary issue is not whether the model is clever enough to resist attack, but whether the surrounding ecosystem can prevent exposed code, shared dependencies, and human mistakes from turning into broad compromise. That is why AI security now intersects directly with identity governance, especially where service accounts, API keys, and AI agents are part of the delivery chain.

The article is pointing to a familiar enterprise pattern: AI systems do not remove human risk, they amplify it through faster data movement and denser integration. In practice, that means access control, secrets handling, and third-party dependency oversight now matter as much for AI as they do for traditional cloud and application estates.

Key questions

Q: What breaks when AI systems rely on exposed code or compromised dependencies?

A: The main failure is not just immediate compromise. Exposed code and tainted dependencies can reveal system design, leak credentials, and give attackers a durable map of how data moves. That means the loss is often permanent, because the attacker learns the control paths as well as the content. Prevention before release matters more than trying to contain the exposure later.

Q: What problem does ownership attribution solve for service accounts and API keys?

A: It closes the gap between exposure detection and accountable remediation. Many organisations can find the secret, but not the human who introduced it, maintains it, or can safely replace it. Ownership attribution gives security teams a practical way to assign action without relying on informal knowledge that disappears during staff changes.

Q: How can security teams tell whether AI lifecycle controls are working?

A: They should look for evidence that access requests, policy enforcement, and usage visibility are centrally recorded and current. If those signals are fragmented across platforms, the programme may be documenting governance rather than enforcing it. Continuous traceability is the practical test.

Q: Who is accountable when AI supply chain exposure leaks customer data or source code?

A: Accountability usually spans product security, application owners, cloud teams, and identity owners because the failure crosses code, dependency, and access boundaries. In regulated environments, the organisation must be able to show that releases, secrets, and third-party dependencies were governed before exposure occurred. Shared responsibility does not remove responsibility.

Technical breakdown

How AI supply chain compromise turns trusted dependencies into exposure

AI applications often depend on open source libraries, APIs, orchestration layers, and hosted services that sit between users and models. When one of those components is compromised, the attacker does not need to break the model itself. They inherit the trust already granted to that dependency and can manipulate data flows, steal secrets, or tamper with outputs. This is classic supply chain risk, but AI increases the blast radius because the dependency may sit inside a high-volume workflow that processes sensitive prompts, credentials, or customer data.

Practical implication: treat AI dependencies as privileged software supply chain assets and review them with the same discipline used for production code and secrets handling.

Why human error becomes a security control failure in AI environments

The Anthropic leak described in the article shows how a release packaging error can expose source code and internal files without any advanced exploitation. Once source code is public, the risk cannot be revoked, which makes prevention far more important than post-incident cleanup. In AI systems, human error matters more because code, prompts, internal workflows, and embedded credentials often converge in one environment. A small packaging mistake can expose design logic, data handling paths, or access patterns that attackers can study for later abuse.

Practical implication: add release checks, content review, and secret scanning to AI deployment pipelines before public exposure can occur.

How AI workflows expand the identity and secrets attack surface

AI tools often rely on API keys, service accounts, and delegated integrations to move data between applications, models, and collaboration platforms. That creates a non-human identity problem as much as an AI problem, because each token or service account can become a pivot point if it is over-permissioned or reused across systems. Once an attacker harvests a secret from an AI supply chain component, they can often move laterally into customer data, admin interfaces, or downstream services without ever touching a user account.

Practical implication: map every AI workflow to the NHI and secret that authorises it, then reduce privilege and revoke standing access where possible.

Threat narrative

Attacker objective: The attacker objective is to turn trusted AI infrastructure into a durable source of credentials, data, and system intelligence for further compromise.

1. Entry occurs through compromised open source dependencies or exposed internal files, which place attacker-controlled code or sensitive artefacts inside the AI delivery chain.
2. Escalation follows when harvested API keys, service accounts, or leaked source code reveal how the environment is wired and where trust boundaries are weak.
3. Impact is achieved when attackers extract customer data, internal system details, or proprietary code that cannot be recovered once exposed.

NHI Mgmt Group analysis

AI security is becoming an identity and trust governance problem, not a model-only problem. The article is correct to frame the risk around how people, tools, and integrations interact with AI systems. That makes the control surface broader than model hardening and closer to IAM, NHI governance, and software supply chain assurance. Practitioners should treat AI systems as trust chains that need continuous identity and access oversight.

Persistent exposure is the key named concept here: once source code or credentials leave controlled boundaries, the loss cannot be undone. That is why the article's prevention-first argument is stronger than a detect-and-contain response model. In AI environments, exposed code can reveal workflows, security assumptions, and data paths in a way that supports follow-on exploitation. Practitioners should focus on blocking irreversible exposure before it reaches production users or shared repositories.

AI supply chain compromise expands the impact of non-human identities. When a dependency such as an orchestration library sits inside an AI workflow, the service accounts and API keys attached to that workflow inherit the compromise. This is exactly where NHI governance becomes relevant to AI security: the credential is not the only asset, but it is often the easiest path in. Practitioners should inventory and constrain every NHI that authorises AI-to-system interaction.

Security awareness alone is no longer an adequate control for AI risk. The article correctly points to human-driven failures, but the operational lesson is that awareness must be backed by enforceable policy, behavioral visibility, and data controls. That aligns with modern governance models that assume mistakes will happen and design to prevent irreversible fallout. Practitioners should move from training-only programmes to control-backed prevention.

AI governance debt is now a practical risk category. Rapid AI adoption often outpaces control design, leaving weak oversight of dependencies, release pipelines, and data-sharing paths. That debt accumulates until a packaging error or supply chain compromise exposes something permanent. Practitioners should measure whether AI governance has been built into change, secrets, and third-party risk management rather than layered on afterwards.

What this signals

Persistent exposure risk: AI incidents increasingly create losses that cannot be reversed because code, secrets, or data are already out of the boundary. That changes the programme objective from detection-led response to prevention-led control design, especially where AI assistants sit inside collaboration and development workflows.

AI governance teams should expect more incidents that start outside the model and end inside the identity layer, where service accounts, API keys, and delegated tokens make compromise portable. The practical response is to tie AI governance to secrets management, access reviews, and third-party risk rather than treating it as a separate programme.

Where AI tools sit on top of Microsoft 365 or similar ecosystems, the real risk is the speed at which trusted users can move sensitive data into places that traditional DLP and app controls do not fully cover. That is why AI security programmes now need identity visibility, not just content inspection.

For practitioners

• Inventory AI dependencies and trust chains Map every library, hosted API, integration, and release step that can move data into or out of AI systems. Prioritise components that can expose credentials, internal files, or customer data if they fail, and track ownership for each dependency.
• Add pre-release exposure controls Require code scanning, secret detection, and packaging validation before AI-related code or artefacts are published. The goal is to stop source code, tokens, and internal files from entering public or broadly accessible locations.
• Map AI workflows to non-human identities Identify which service accounts, API keys, certificates, and delegated tokens authorise each AI workflow. Remove shared credentials, shorten credential lifetime, and review whether standing access is still justified for every downstream integration.
• Tighten data controls across collaboration and AI channels Review how sensitive data moves through email, collaboration apps, and AI assistants, then enforce policy on what can be shared, copied, or forwarded into AI tools. Pair data loss prevention with access visibility so risky transfers are blocked rather than only logged.

Key takeaways

• AI security failures are increasingly caused by human error and supply chain compromise rather than model weakness alone.
• The evidence points to permanent exposure risk, because leaked code and harvested secrets cannot be recalled once they leave controlled boundaries.
• Practitioners should govern AI as a trust chain, tying dependency review, secrets management, and NHI oversight to every workflow.

Key terms

• AI supply chain: The AI supply chain is the full chain of models, datasets, prompts, tools, and vendors that influence a deployed AI system. It matters because trust cannot be assigned to the application alone. Practitioners need provenance, ownership, and dependency visibility to govern risk.
• Persistent Exposure: A security condition where code, data, or credentials cannot be fully recovered once they have been exposed outside the intended boundary. In AI environments, persistent exposure is especially dangerous because leaked material can reveal system design, access paths, and downstream data handling.
• Non-Human Identity (NHI): A digital identity assigned to a non-human entity such as a software application, service account, API key, bot, machine, or AI agent that enables it to authenticate and interact with systems without direct human involvement. NHIs now outnumber human identities in most enterprises by 25 to 50 times.
• AI Governance: AI governance is the set of controls used to discover, classify, approve, restrict, monitor, and revoke AI-enabled access. It connects identity, data, and policy so organisations can manage what AI can reach, what it can share, and when it should be stopped.

What's in the full article

Proofpoint's full article covers the operational detail this post intentionally leaves for the source:

• Specific examples of how AI-related exposure happened in the Anthropic leak and Mercor supply chain case.
• The article's breakdown of why release packaging and open source dependency risk behave differently in AI environments.
• Practical examples of how AI security failures intersect with Microsoft 365, collaboration platforms, and downstream data sharing.
• Proofpoint's recommended control priorities for reducing human-driven AI exposure and supply chain risk.

👉 Proofpoint's full post covers the human, dependency, and data paths that shaped these incidents.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, workload identity, and agentic AI identity. It is designed for practitioners who need to connect identity controls to modern application and AI risk.