Who is accountable when AI supply chain exposure leaks customer data or source code?

Why This Matters for Security Teams

AI supply chain exposure is rarely a single-team problem. A leaked customer dataset or source code repository can originate in model training pipelines, CI/CD, dependency management, container images, secrets handling, or a third-party service with overbroad access. The accountability question matters because incident response, legal exposure, customer notification, and remediation all depend on being able to trace where trust broke down and who owned the control at that point.

Security teams often underestimate how quickly AI systems expand the blast radius. A model may not be the direct source of the leak, but it can still surface sensitive data through retrieval paths, logging, prompt handling, plugin integrations, or agent tool access. That makes governance of Non-Human Identity, service accounts, tokens, and automated build identities part of the same risk chain as code security. Guidance from the OWASP Non-Human Identity Top 10 is especially relevant here because many exposures begin with credentials and machine-to-machine trust that were never reviewed as rigorously as human access.

In practice, many security teams encounter ownership gaps only after a leaked secret, dependency compromise, or data exfiltration has already reached customers.

How It Works in Practice

Accountability should be assigned across the lifecycle, not after the incident. Product security typically owns secure design and release governance. Application owners own the business use case and data flows. Cloud or platform teams own the environment controls, including logging, network restrictions, and secret distribution. Identity owners own access policy, service account lifecycle, and privileged delegation. When an AI system is involved, these roles need explicit boundaries because the same exposure can traverse source control, model orchestration, and runtime access in a single workflow.

In practice, teams should map the path of sensitive data and code through the system, then anchor each stage to a named control owner. That includes:

• Inventorying models, datasets, connectors, plugins, and build dependencies.
• Classifying secrets, customer data, and source code as protected assets with approved handling rules.
• Restricting service account and agent privileges to the minimum required for each workflow.
• Logging access to repositories, artifact stores, training data, and inference endpoints.
• Reviewing third-party and open-source components for provenance, integrity, and update hygiene.

For AI-specific environments, the question is not only who deployed the model, but who approved the data sources, tool permissions, and fallback controls. The NIST IR 8596 Cyber AI Profile is useful because it frames AI risk in operational terms, including governance, measurement, and response. Pair that with the control discipline in NIST SP 800-53 Rev 5 Security and Privacy Controls to ensure access, auditing, and configuration management are not treated as optional extras.

These controls tend to break down when AI pipelines are assembled from ephemeral services and unmanaged machine identities because ownership becomes fragmented across teams and vendors.

Common Variations and Edge Cases

Tighter control over AI supply chains often increases release friction, requiring organisations to balance speed against assurance. That tradeoff is real, especially for teams shipping frequent model updates or relying on external APIs, but current guidance suggests that the answer is not looser governance. It is clearer accountability, documented exceptions, and enforceable approval paths.

There is no universal standard for shared accountability wording yet, so contracts, internal policy, and technical control evidence all matter. If a vendor-hosted model or managed agent handled the exposed data, responsibility may be shared, but the organisation still remains accountable for selecting the service, configuring access, and monitoring outcomes. Where source code leakage is involved, developer tooling, Git permissions, CI service principals, and artifact registries need to be examined together, not as isolated systems.

Edge cases become especially difficult when autonomous agents can read repositories, call external tools, or create pull requests. In those environments, the boundary between human approval and machine execution narrows, so identity governance must extend to non-human actors. That is where AI security and NHI governance intersect most sharply, and where failure often appears as an access problem long before it is recognised as a data breach.

When incident reviews stop at the last team that touched the pipeline, the real control failure is usually missed.

Related resources from NHI Mgmt Group

• Who should be accountable when an AI marketing agent changes customer data incorrectly?
• Who is accountable when sensitive data leaks through consumer AI tools?
• Who is accountable when an AI agent in a pipeline leaks credentials and enables code push access?
• Who is accountable when an AI-assisted workflow leaks sensitive data?