How should security teams measure AI risk in software they actually ship?

What to measure when AI is shipping inside real software

The right unit of measurement is the shipped asset, not the model demo. Security teams need inventory counts that answer whether AI is present in production code, whether it is sanctioned, who owns it, and whether the surrounding configuration contains secrets or other sensitive runtime material. That shifts AI risk from abstract model quality to the operational surface where compromise, misuse, and governance gaps actually appear.

Counts also give you a baseline you can trend. If the number of unapproved AI assets rises faster than the number of reviewed ones, the problem is not model drift, it is control drift. If ownerless AI assets keep showing up, you have a governance failure that will make review, exception handling, and incident response slower.

A useful way to frame this is to measure reachability and control together: what exists, what is allowed, what is owned, and what can be abused. That includes AI configuration files, deployment manifests, orchestration settings, and any file that can leak tokens, keys, or service credentials into the runtime path. In that sense, the inventory is not just a catalog, it is a map of exposure.

Why shipped AI risk is mostly code, tooling, and credentials

Model behavior matters, but it is rarely the whole risk story in shipped software. A safe model embedded in a weak integration can still create material exposure if the surrounding service has broad permissions, long-lived secrets, or unreviewed dependencies. The practical question is whether the AI feature can reach data, call tools, or act on behalf of the organization beyond what the business intended.

This is why security teams should count secrets in AI configuration files and adjacent build or deployment artifacts, not just look for prompt quality issues. When credentials sit close to the AI control plane, the blast radius is often larger than the model itself because compromise can convert into lateral access, data exposure, or unauthorized actions through whatever the secret unlocks.

It is also important to distinguish approved AI from merely visible AI. Unapproved components are often the highest-risk items because they bypass normal architecture review, logging expectations, and ownership assignment. Once AI is embedded in shipping code, the relevant control question becomes whether the asset is observable and governed enough to prevent silent expansion of privilege.

For broader context on how teams structure this problem, the AI Security Platform Buyer’s Guide is useful for comparing controls that cover inventory, guardrails, and runtime oversight. For shipped AI systems, the AI Infrastructure Workload Identity Guide is a strong companion when the key question is which pipelines, notebooks, registries, and inference paths can authenticate and act at runtime.

How to turn AI counts into a decision-grade risk signal

Raw totals are only useful if they separate exposure from noise. The first practical split is between sanctioned and unsanctioned AI, because that tells you where governance is working and where shadow deployment is bypassing review. The second split is between owned and ownerless AI, because ownership is what makes remediation, exception approval, and incident escalation possible.

Security teams should then look for concentration, not just volume. A small number of AI assets with broad access, shared credentials, or cross-environment reach can matter more than a larger number of low-impact assets. That is why counts should be paired with privilege and placement, so you can see whether one service credential or one misconfigured integration can unlock too much.

The most decision-useful view is usually a simple matrix: asset count, approval status, named owner, secret exposure, and scope of access. When all four are visible, you can prioritize by blast radius instead of debate. That also helps separate hygiene problems, like stray configuration secrets, from architectural problems, like AI services that can reach production systems without enough constraint.

The same pattern appears in the Agentic AI Security Guide, which maps runtime controls to identity, tools, and orchestration, and in the Top 10 Agentic AI Identity Issues, which is especially helpful when shipped software lets an agent act with privileges that were never intended for routine use.

Risk and Threat Considerations

Shipped AI creates risk when inventory and authority drift apart. The danger is not only that an AI feature exists, but that it exists without a clear owner, with unreviewed secrets, or with enough access to become a high-impact pivot point if compromised. That makes the control problem cumulative: every unmanaged asset increases the chance that a future incident becomes a broader one.

Failure mechanism: Attackers and internal abuse paths exploit the gap between visibility and governance, for example by targeting unapproved AI services, harvesting secrets from configuration files, or using over-permissive runtime credentials to reach data and tools beyond the intended scope.

Impact: The result can be unauthorized action, data exposure, lateral movement, and much larger blast radius than a model-only review would suggest. In mature environments, these are the failures that turn an AI feature into an enterprise incident.

Practitioner Guidance

What to prioritise: Start with a complete inventory of shipped AI assets, then rank them by approval status, named ownership, and secret exposure. If an asset cannot be owned or explained, treat it as a governance problem first and a model-risk problem second.

What to verify: Confirm that every AI component in production has a named accountable owner, a documented purpose, and a reviewed path for secrets. If the same credential appears in multiple AI files or environments, assume the blast radius is already larger than the architecture diagram suggests.

Practitioner takeaway: Measure AI risk by the controls around shipped software, not by the model’s output alone, because the operational question is whether AI can be reached, governed, and constrained before it can be abused.

Related resources from NHI Mgmt Group

• How should security teams handle risks from AI browser extensions?
• How should security teams govern API keys used for generative AI access?
• How should security teams measure whether AI is helping rather than hiding risk?
• How should security teams measure whether identity governance is actually reducing risk?