AI bill of materials gaps are obscuring production model risk

At a glance

What this is: Cycode argues that AIBOMs are meant to close the gap between what AI teams think is running and what is actually deployed, because production model risk is often hidden across tools, teams, and spreadsheets.

Why it matters: For IAM, NHI, AI governance, and security teams, the lesson is that visibility must move from periodic documentation to continuous control if they want to govern models, prompts, dependencies, and approvals at production speed.

By the numbers:

• Only 19% of organizations have full visibility into where and how AI is used across development.

Context

An AI bill of materials is a governance inventory for AI systems, covering models, datasets, prompts, dependencies, and controls across the lifecycle. The visibility problem it addresses is not abstract: production AI changes quickly, yet many organisations still rely on split ownership, spreadsheets, and disconnected tooling to understand what is actually deployed.

That gap matters because AI supply chains now include third-party models, open-source dependencies, and continuously changing prompts and datasets. When the inventory is stale or incomplete, teams lose the ability to prove provenance, enforce policy, or explain model behaviour when auditors, regulators, or incident responders ask.

The article’s starting position is typical of fast-moving AI programmes: adoption outruns governance unless discovery and control are automated.

Key questions

Q: What breaks when AIBOM coverage is still manual?

A: Manual AIBOMs go stale as soon as models, prompts, datasets, or dependencies change outside the inventory process. That creates blind spots for governance, auditability, and change approval. The practical failure is not only missing documentation, but the loss of a reliable control point for deciding whether the production AI state is still authorised.

Q: Why do AI provenance gaps create security risk for enterprises?

A: They remove the evidence needed to separate authorised automation from shadow AI, data leakage, and disputed model output. If users can place sensitive material into external tools without a trace, the organisation loses control over confidentiality and accountability. Provenance gaps are therefore both a security and governance problem.

Q: How do teams know if an AIBOM is actually working?

A: An AIBOM is working when auditors can trace each production AI system back to its model version, dataset lineage, prompt set, dependency chain, and approval history without manual reconstruction. If that trace requires spreadsheets or ad hoc interviews, the AIBOM is not yet functioning as a control.

Q: When should organisations prioritise inventory over broader AI governance controls?

A: Inventory should come first because every downstream control depends on knowing what exists. Risk classification, assessments, access review and lifecycle management all assume the asset is already identified. If the record is missing or stale, the organisation is governing a partial estate and cannot trust the rest of the programme.

Technical breakdown

Why manual AIBOMs go stale in CI/CD pipelines

An AIBOM only works if it tracks change at the same pace as the delivery pipeline. In modern AI development, models are retrained, datasets are swapped, prompts are refined, and dependencies shift multiple times per day. A static inventory quickly becomes a snapshot of yesterday’s system, which is why manual documentation tends to miss shadow AI assets, unapproved model substitutions, and policy drift. The operational problem is not the idea of an inventory, but its inability to stay synchronised with continuous integration, deployment, and runtime change.

Practical implication: Treat AIBOM freshness as a control objective, not a documentation task.

How model provenance and dataset lineage reduce AI supply chain risk

Provenance means being able to show where a model or dataset came from, how it changed, and who approved those changes. That matters because AI risk is often introduced through invisible supply chain paths such as unvetted training data, open-source libraries, and third-party APIs. Without lineage, teams cannot distinguish sanctioned updates from tampering, licensing violations, or corrupted inputs. In governance terms, provenance becomes the evidence layer that supports auditability, reproducibility, and trust in production AI.

Practical implication: Track source, version history, and approval checkpoints for every production AI component.

Why policy enforcement has to move into the development workflow

AIBOMs are most useful when they do not sit beside the workflow but inside it. The article’s core argument is that governance should be enforced through pull requests, build gates, and deployment checks, rather than through after-the-fact review. That is especially important when AI components include prompts, fine-tunes, and dependencies that can alter behaviour without a visible application change. By tying policy to the workflow, organisations create an auditable path from change request to deployed model state.

Practical implication: Embed AI policy checks into pull requests and release gates before non-compliant components reach production.

Threat narrative

Attacker objective: The objective is to get unverified or manipulated AI components into production without detection, then exploit the resulting governance blind spot.

1. Entry occurs when AI components are added or modified through normal development activity, including new datasets, prompt changes, third-party services, or retrained models that are not fully inventoried.
2. Escalation happens when those changes are not captured in a current AIBOM, allowing unapproved or risky components to inherit production trust without review.
3. Impact follows when a stale inventory prevents teams from spotting drift, provenance issues, or supply chain compromises before the AI system influences decisions in production.

Breaches seen in the wild

• LiteLLM PyPI package breach: LiteLLM PyPI supply chain attack, credentials stolen from users.
• Nx s1ngularity attack 2025: Attackers stole Nx's npm token via a GitHub Actions flaw and shipped malware that stole 2,349 secrets and abused developers' AI CLIs.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Visibility debt is now an AI governance problem, not a documentation problem. The article shows that the main failure is not absence of policy language but absence of continuous inventory across models, datasets, prompts, dependencies, and controls. When governance depends on manual records, it always trails the production system. For practitioners, the real question is whether the organisation can prove what is running before the next model change lands.

AI supply chain risk is becoming a provenance problem. Once model updates, dataset substitutions, and third-party APIs move through the pipeline, teams need traceability that is strong enough to separate authorised change from silent drift. That maps to broader supply chain governance and, where identity or access is involved, to tighter control over who can alter AI components. The practitioner conclusion is that provenance must be treated as a security control, not a post-incident report.

Continuous discovery is the only workable control model for fast-changing AI estates. The article makes clear that AIBOMs fail when they are treated as static artefacts, because AI systems are modified too often for periodic review to be reliable. This is where the identity bridge matters: if the people, services, or pipelines that change AI assets are not governed with strong access and approval boundaries, the inventory will never stay current. Practitioners should align discovery, approvals, and runtime monitoring as one control loop.

AIBOM governance exposes a broader control gap that existing SDLC and SBOM practice does not close. Traditional software inventories describe code dependencies well, but they do not automatically capture prompt behaviour, model lineage, or runtime AI controls. That gap is why AI governance needs its own operating model even when it connects to SBOM and ASPM. The field should expect AIBOM to become a baseline expectation for accountable AI use, not an optional reporting layer.

From our research library:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap, according to the State of Secrets in AppSec.
• Read next: AI Supply Chain Security and AI-BOM Guide

What this signals

AI bill of materials debt is emerging as a governance gap that looks small until production changes start piling up. As model updates, prompt edits, and dataset swaps accelerate, the inventory becomes the control plane that determines whether the organisation can still explain what it has deployed. For practitioners, that means discovery and approval must be treated as operational controls, not back-office documentation.

Continuous AI inventorying is now part of broader security posture management. The article’s message aligns with a shift from periodic audit prep to always-on visibility across AI assets, dependencies, and change history. Where model provenance or approval state is unclear, the safer assumption is that risk is already present in the pipeline.

Model provenance gap: when AI assets can change without a reliable trace, governance cannot distinguish authorised iteration from silent drift. That weakens auditability and makes blast-radius assessment unreliable, so security teams need tighter linkage between development workflows and runtime oversight.

For practitioners

• Automate AI asset discovery Continuously identify models, datasets, prompts, dependencies, and third-party AI services across repositories, build systems, and production environments.
• Define ownership for every AI component Assign a named owner for each model and dataset so approvals, updates, and exceptions have clear accountability.
• Move policy checks into CI/CD gates Enforce AIBOM policy during pull requests, merge workflows, and deployments so non-compliant AI components do not reach production.
• Track provenance and approval history Record source, version history, training data, and approval checkpoints for each AI asset to support audit and incident response.
• Correlate AI risk with SBOM and ASPM Use a unified view to connect AI components, software dependencies, and infrastructure risk so remediation prioritises the highest blast radius.

Key takeaways

• AI governance fails fastest when teams cannot prove which models, datasets, prompts, and dependencies are actually in production.
• Cycode cites only 19% full visibility into where and how AI is used across development, underscoring how common the inventory gap remains.
• The control answer is continuous discovery plus workflow-enforced policy, because manual AIBOMs cannot keep pace with modern AI change rates.

Key terms

• AI bill of materials: An inventory of the components used by an AI development environment, including models, plugins, hooks, skills, and agent integrations. It gives security teams visibility into what is operating in the stack so they can assess trust, dependency risk, and governance coverage.
• Model Provenance: Model provenance is the evidence chain showing where an AI artefact came from, how it was modified, and whether the version in use is the one that was approved. For AI security teams, provenance is the control that turns trust from assumption into verification.
• Data Lineage: The record of how data moves across systems, applications, and workflows. In security operations, lineage shows where sensitive data propagates, which identities touch it, and how a compromise could spread across connected environments.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and the control thinking that underpins reliable inventory and approval models. It helps security and identity practitioners connect governance design to the access and lifecycle decisions that keep production systems accountable.