What breaks when organisations cannot inventory GenAI usage in their codebase?

Why This Matters for Security Teams

When GenAI is introduced into a codebase without a reliable inventory, security teams lose the most basic control point: knowing what exists. That makes it difficult to assign ownership, review dependencies, or decide whether a model call is experimental, embedded, or production-critical. It also weakens policy enforcement because the security team cannot distinguish sanctioned use from shadow use.

This is not just a documentation problem. The NIST AI 600-1 GenAI Profile treats inventory, traceability, and governance as part of operational risk management, and NHIMG research shows why that matters in practice: in DeepSeek breach, secrets and exposed systems created a far wider blast radius than a single application team could easily track. Without inventory, remediation becomes reactive, and risk decisions are made after exposure has already spread across repositories, environments, and service accounts.

In practice, many security teams first discover uncontrolled GenAI usage only after a dependency breaks, a prompt path leaks data, or an incident review reveals the code path was never formally approved.

How It Works in Practice

A workable inventory does more than list model names. It records where GenAI appears in source code, infrastructure-as-code, build pipelines, and runtime integrations, then ties each instance to an owner, a purpose, and a data classification. That gives security teams a way to separate a proof of concept from a customer-facing feature and to understand which controls should apply at each stage.

For codebase review, teams usually combine static scanning, dependency inspection, API usage detection, and developer attestations. The goal is to identify hard-coded model endpoints, SDKs, orchestration libraries, prompt templates, tool integrations, and any secrets used to reach external model services. A useful inventory also tracks whether the application stores prompts or outputs, because those artifacts may contain credentials, personal data, or regulated content.

Operationally, this should feed change management and risk review. If a team adds a GenAI feature, the inventory should trigger an owner assignment, a review of data flows, a check for secret handling, and a decision about whether the workload belongs in development, test, or production. The same discipline applies to vendor-hosted and self-hosted models, because both can introduce hidden dependencies.

• Map every GenAI touchpoint to a system owner and business purpose.
• Classify the data the model can see, store, or generate.
• Track dependencies on model providers, SDKs, prompts, and tool connectors.
• Record secrets, tokens, and service accounts used by the application.
• Reconcile inventory entries against deployment and CI/CD changes on a set cadence.

Current guidance suggests aligning this with the State of Secrets in AppSec, because GenAI inventory gaps often overlap with weak secret hygiene and poor code visibility. These controls tend to break down in fast-moving CI/CD environments where teams can ship new prompts, plugins, or model calls between scans and without formal architecture review.

Common Variations and Edge Cases

Tighter GenAI inventory controls often increase engineering overhead, requiring organisations to balance visibility against delivery speed. That tradeoff becomes sharper when GenAI is embedded through third-party libraries, internal platforms, or feature flags, because the code may not obviously look like an AI application even though it behaves like one.

One common edge case is indirect usage: a service may not call a model directly, but it may invoke another internal service that does. Another is shared tooling, where one prompt library or orchestration layer is reused across many repositories, making ownership ambiguous unless the inventory includes dependency lineage. Best practice is evolving here, and there is no universal standard for what level of granularity is sufficient.

Another blind spot is the difference between detection and governance. Finding a model string in code is not enough if the organisation cannot tell whether that usage is sanctioned, what data it touches, or whether the surrounding controls are current. The Schneider Electric credentials breach reinforces the point that exposed code paths and exposed credentials often move together, so inventory needs to capture both application logic and the identity material around it.

In practice, the hardest cases are legacy monoliths, AI features hidden in shared libraries, and multi-team platforms where no single owner can confirm whether GenAI is still in use.

Related resources from NHI Mgmt Group

• What breaks when organisations cannot inventory cryptographic libraries and algorithms in their products?
• What breaks when organisations cannot continuously inventory non-human access across apps and repositories?
• What breaks when organisations cannot see how access is granted through roles, groups, and trust relationships?
• What breaks when organisations cannot trace data from source to report or model?