Shadow AI in the software supply chain is breaking AppSec visibility

At a glance

What this is: This is an AppSec analysis of shadow AI in the software supply chain, showing that models, agents, datasets, prompts, and MCP servers are entering codebases faster than existing tooling can inventory them.

Why it matters: It matters because AppSec, IAM, and AI governance teams now need deterministic visibility into AI components that behave like supply chain dependencies and can introduce identity, access, and compliance risk.

By the numbers:

• Over 75% of organizations have experienced a software supply chain attack in the last year.

👉 Read Checkmarx's analysis of shadow AI in the software supply chain

Context

Shadow AI is the growing use of models, agent frameworks, MCP servers, datasets, prompts, and related components without formal security review. The problem is not simply more software, but a parallel supply chain that existing AppSec tools often cannot see or classify.

That creates a governance gap for AppSec and adjacent identity programmes. Once AI components can invoke tools, access data, or embed secrets in code and config, they start to look less like static dependencies and more like governed runtime actors that need lifecycle control.

Key questions

Q: How should security teams govern shadow AI in SaaS environments?

A: Security teams should inventory AI-enabled features, classify the data those features can touch, and enforce approved-use rules at the application and identity layers. The practical goal is not to block every model interaction. It is to ensure that prompts, file uploads, and connected data sources stay within defined risk boundaries.

Q: Why does AI make software supply chain risk harder to control?

A: AI increases the amount of code produced, which reduces the time available for review and makes malicious or unauthorized changes harder to spot. It also introduces non-human actors into the development flow, so traditional assumptions about developer identity no longer hold. That combination expands the attack surface at the commit stage.

Q: What breaks when AI agents are not inventoried or classified?

A: When AI agents are not inventoried or classified, organisations lose the ability to assign risk, apply the right obligations, and prove control to auditors. The result is hidden access, inconsistent oversight, and weak accountability when the agent touches regulated data or triggers business actions.

Q: How can organisations prove their AI controls are actually working?

A: Look for evidence that policy decisions are logged, sensitive prompts are being redacted or blocked when required, and approved AI interactions are traceable by identity and business context. Effective programmes produce audit-ready records, not just policy text. If the control cannot explain what happened in a session, it is not operational enough.

Technical breakdown

Why AI components escape traditional SBOM and SCA coverage

Standard software composition analysis was built to identify code libraries and package dependencies, not the broader AI stack. Models from public hubs, prompt libraries, embeddings, dataset references, and MCP server configurations often live in files and paths that do not map cleanly to a dependency graph. That means traditional inventories can miss the very artefacts that shape application behaviour, data exposure, and downstream trust. In practice, the challenge is not just discovery, but classification: security teams need to know what is a model, what is a tool connection, and what is a data reference before they can apply policy.

Practical implication: extend inventory controls beyond manifests so AI artefacts are discovered as first-class assets.

Deterministic discovery versus probabilistic AI detection

A deterministic approach reads AI assets directly from imports, initialization code, manifests, file paths, and configuration entries. That matters because AI inference based detection introduces uncertainty, and uncertainty is a poor basis for audit, gating, or compliance evidence. Deterministic discovery can prove that a specific model reference, agent framework, or MCP server exists in a codebase and trace it back to a line of code or config. For AppSec teams, that evidence chain is what turns AI governance from a best effort into something that can be reviewed, repeated, and defended.

Practical implication: prefer code- and config-based discovery for AI assets when auditability matters.

AI supply chain risk is broader than vulnerable code

AI supply chain risk includes model poisoning, unverified weights, unpinned versions, unsafe autonomous agents, exposed datasets, and licence violations. These are not just implementation defects. They are governance failures that can alter application behaviour, expose sensitive data, or create unauthorised execution paths. The presence of agent frameworks and MCP servers raises the stakes further because tool invocation can turn a software dependency into a control plane issue. That shifts the question from whether the code scans cleanly to whether the AI component can be trusted to act within approved boundaries.

Practical implication: assess AI components for trust, versioning, and tool-use boundaries, not just code defects.

Threat narrative

Attacker objective: The objective is to exploit unseen AI components to gain data access, manipulate behaviour, or create a durable supply chain foothold inside the application stack.

1. Entry occurs when developers import pre-trained models, agent frameworks, or MCP servers into applications without formal review or inventory.
2. Escalation happens when those components can invoke tools, access datasets, or carry unverified model content into production workflows.
3. Impact follows when unmanaged AI supply chain components expose sensitive data, alter application behaviour, or create an audit gap during investigation.

NHI Mgmt Group analysis

Shadow AI is becoming a supply chain governance problem, not just an AppSec visibility issue. Once models, prompts, MCP servers, and datasets are embedded in code, they function like operational dependencies with security implications. Traditional SCA and SBOM practices were not designed to capture those relationships, so the governance gap is structural rather than incidental. Practitioners should treat AI artefacts as governed assets from the moment they enter the build system.

Deterministic discovery is the right control model for audit-grade AI inventory. Probabilistic detection may be useful for triage, but it is weak evidence when a team must prove what is present, where it lives, and how it is wired. A code-first approach creates a defensible chain from configuration to finding to remediation. That matters because auditability now sits alongside detection as a core control objective.

AI governance debt is the right name for the accumulation of unreviewed AI components. Each unpinned model, embedded prompt, or unmanaged MCP server adds future remediation cost and expands the blast radius of a build pipeline. The longer those assets remain invisible, the harder it becomes to apply policy consistently across code, CI/CD, and runtime. Practitioners should assume the debt compounds faster than conventional vulnerability backlogs.

Agent frameworks and MCP servers blur the line between dependency and delegated access. When software can call tools autonomously, the security question is no longer only whether the package is safe, but whether the execution path is authorised. That creates a real intersection with identity governance, because runtime trust must extend to the non-human components acting on behalf of the application. Teams should align AI supply chain controls with least privilege and tool-scoping discipline.

Compliance pressure will force AI component governance into mainstream AppSec operations. The article's references to the EU AI Act, ISO 42001, and NIST AI RMF reflect a broader shift from optional oversight to accountable control design. Organisations will need repeatable evidence for inventory, policy enforcement, and change control if they want their AI programmes to survive scrutiny. Practitioners should plan for AI governance to be assessed like any other material risk domain.

What this signals

Shadow AI is now a governance problem that sits between AppSec, platform engineering, and identity teams. The operational signal is clear: if AI artefacts cannot be inventoried deterministically, policy enforcement will stay reactive and exceptions will keep accumulating. That is why the next control boundary is not just scanning code, but proving what non-human components are authorised to act inside the software supply chain.

AI governance debt: the backlog created when unreviewed models, prompts, MCP servers, and datasets accumulate faster than teams can classify them. The longer that debt remains, the more expensive every remediation becomes because the security team must untangle code, data, and delegated execution at once. Practitioners should expect AI BOM discipline to become a routine expectation in build governance, not a specialist add-on.

For teams mapping this work to broader standards, the most relevant control lens is the NIST AI Risk Management Framework combined with OWASP guidance for agentic applications. That combination helps translate discovery into accountability, policy, and monitoring without forcing AI risk into a pure vulnerability-management model. The practical next step is to connect approvals, evidence, and runtime exceptions into one chain that auditors and engineers can both follow.

For practitioners

• Inventory AI artefacts as first-class assets Track models, agent frameworks, MCP servers, datasets, embeddings, and prompt files in the same governance process used for software dependencies. Tie each asset to code location, ownership, and approval status so the build pipeline can fail closed when an unapproved component appears.
• Enforce deterministic discovery in CI/CD Use code and configuration signals to identify AI dependencies instead of relying on inference-based detection. Require findings to map back to specific imports, manifests, file paths, or config entries before they can enter triage.
• Gate unpinned models and unsafe agents Block floating model references, unverified weights, and autonomous tool use without explicit scope constraints. Treat version pinning and tool allowlisting as release gates, not post-deployment cleanup tasks.
• Extend policy enforcement into pull requests Apply approval rules before AI components merge, rather than after they ship. Connect the same policy checks used for vulnerable packages to AI artefacts so developers get one workflow instead of parallel exception handling.
• Map AI governance to compliance evidence Align AI-BOM output, remediation records, and approval trails to the controls required by the EU AI Act, ISO 42001, and NIST AI RMF. Make sure auditors can see what was present, what was blocked, and what was accepted with risk.

Key takeaways

• Shadow AI turns the software supply chain into a governance problem because models, prompts, MCP servers, and datasets now change application behaviour as well as code content.
• The evidence points to a growing visibility gap, with AI-generated code creating 1.7x more vulnerabilities and most organisations still unable to track AI agent data access end to end.
• Practitioners need deterministic discovery, AI-BOM discipline, and CI/CD policy gates if they want AI governance to be auditable rather than aspirational.

Key terms

• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• AI-BOM: An AI bill of materials is a structured inventory of the components that define an AI agent, including the model, prompt, tools, retrieval sources, and dependencies. In practice, it is the evidence base for review, change control, and risk assessment when the agent evolves after deployment.
• Deterministic discovery: Deterministic discovery is a method of identifying AI assets from explicit code and configuration signals such as imports, manifests, file paths, and settings. It is preferred for governance because it creates repeatable, auditable evidence rather than relying on probabilistic inference.
• MCP Server: An MCP server is a tool endpoint that connects an AI agent to external systems and data sources through Model Context Protocol. Because it extends what the agent can reach, it becomes part of the identity and access surface and must be reviewed like any other privileged connector.

What's in the full article

Checkmarx's full article covers the operational detail this post intentionally leaves for the source:

• How the code-first discovery workflow identifies models, agent frameworks, MCP servers, datasets, and prompts from imports and configuration.
• Which AI-specific risk checks apply to unverified weights, floating model references, unsafe autonomous agents, and exposed datasets.
• How AI-BOM output fits into existing AppSec dashboards, CI/CD gates, and policy workflows without a separate platform.
• Which compliance mappings support EU AI Act, ISO 42001, NIST AI RMF, and OWASP LLM Top 10 reporting.

👉 The full Checkmarx article covers the deterministic discovery model, AI-BOM workflow, and policy enforcement details.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is useful for practitioners who need to connect identity controls to the broader governance of non-human systems.