What breaks when organisations rely on SBOMs alone for AI-enabled applications?

Why This Matters for Security Teams

SBOMs were designed to improve software transparency, but AI-enabled applications introduce components that are not captured by a conventional package inventory. Models, agents, prompts, retrieval layers, fine-tuning data, and tool connectors can all change application behaviour without altering the dependency tree. That means a team can have a neat software list and still lack evidence for what the system can do, what inputs shape its outputs, and which artefacts were approved for production. This is a governance problem as much as a technical one, and it maps closely to the risk themes in the NIST Cybersecurity Framework 2.0.

The common mistake is treating the SBOM as the control itself rather than as one input to a broader assurance process. For AI-enabled applications, security teams need to know provenance, lineage, ownership, and operational authority. Without that, incidents become difficult to triage and audits become shallow because evidence stops at the package boundary. In practice, many security teams encounter AI governance gaps only after a model, agent, or connector has already been deployed without a reliable approval trail.

How It Works in Practice

An SBOM can still be useful, but only as the software layer of a wider inventory. For AI-enabled applications, organisations usually need to pair it with model documentation, data lineage records, prompt and policy controls, and an asset register for agents and MCP servers. Current guidance suggests that reviewers should be able to trace not just what code is present, but also what model version is in use, where it came from, which datasets influenced it, and which tools it can invoke.

That broader view is closer to the assurance logic used in NIST AI Risk Management Framework and related AI governance guidance. In operational terms, teams should maintain:

• an inventory of models, agents, prompts, RAG sources, and external tool integrations;
• provenance records for training, fine-tuning, and retrieval data;
• approval and ownership records showing who authorised deployment and what constraints apply;
• validation evidence covering expected outputs, safety filters, and fallback behaviour;
• change control for model updates, prompt revisions, connector changes, and policy edits.

This is also where AI-specific threat models matter. SBOMs do not show prompt injection paths, model poisoning risk, or tool misuse by autonomous agents. Security teams therefore need testing and monitoring aligned to adversarial AI techniques, including the patterns described in MITRE ATLAS. Where AI systems are connected to identity or privileged workflows, the control question becomes whether the entity acting on behalf of the system has explicit, bounded authority rather than implicit trust. These controls tend to break down in fast-moving MLOps environments with frequent model swaps, unmanaged prompt changes, and shadow AI connectors because the inventory cannot keep pace with runtime behaviour.

Common Variations and Edge Cases

Tighter inventory and provenance controls often increase operational overhead, requiring organisations to balance assurance against deployment speed. That tradeoff is especially visible when teams use managed model APIs, third-party embeddings, or ephemeral agents that call internal services on demand. There is no universal standard for this yet, so best practice is evolving rather than fixed.

In regulated environments, the gap between an SBOM and AI assurance becomes more serious. Under the EU AI Act, evidence expectations can extend beyond software composition into risk management, documentation, and human oversight. Where AI-enabled applications touch personal data, critical workflows, or regulated decision-making, teams may also need stronger records for explainability, data governance, and operational accountability. For many organisations, the real edge case is not a large model in a lab, but a small AI feature embedded inside a business system where no one treats it like a distinct asset.

SBOMs should therefore be treated as necessary but incomplete. They help identify one layer of supply-chain risk, but they do not replace model registers, AI governance controls, or runtime monitoring. Organisations that rely on SBOMs alone usually discover the blind spot when a reviewer asks who approved the model, where its data came from, or what authority the agent had to act.

Related resources from NHI Mgmt Group

• What breaks when organisations rely on endpoint controls alone for AI use?
• What breaks when organisations rely on human oversight alone for AI risk?
• What breaks when organisations rely on DMARC alone against AI-driven phishing?
• What breaks when organisations rely on legacy DLP for AI workflows?