Why do model files create more risk than most teams expect in AI supply chains?

Why This Matters for Security Teams

Model files are not just data blobs. In practice, they sit inside a software supply chain that often trusts artifacts before it inspects them. A malicious checkpoint can trigger code execution through deserialization, custom operators, or embedded payloads, which means the compromise can occur before normal application defenses, logging, or sandboxing are fully active. That shifts the risk from “bad model output” to “host compromise at load time.”

This matters because teams often inherit legacy software assumptions that artifacts are passive inputs. They are not. A model pulled from a public registry or a third-party package can behave more like an executable than a document, especially when the loading stack includes native extensions or convenience wrappers. Guidance from the OWASP Non-Human Identity Top 10 and NIST Cybersecurity Framework 2.0 both point to the same operational reality: trust boundaries must move closer to the artifact source and the identity of the actor delivering it.

NHIMG research shows why this is not a theoretical concern. The Mastra npm Supply Chain Attack and LiteLLM PyPI package breach show how AI-adjacent packages and model tooling can be abused to reach secrets and downstream systems. In practice, many security teams discover this only after a model loader has already executed untrusted code, rather than through intentional artifact review.

How It Works in Practice

The core mistake is treating model distribution like static file delivery instead of software delivery. Model formats such as pickle-based artifacts, unsafe serialization wrappers, and plugin-enabled runtimes can execute logic when loaded. Even when the model itself is harmless, dependencies around it may not be. That includes pre/post-processing code, native kernels, custom operators, and registry-side metadata that influences execution paths.

Security teams should evaluate model artifacts the same way they evaluate third-party code: provenance, signing, integrity checks, and restricted execution environments. Current guidance suggests combining artifact verification with least privilege and isolated loading paths, because runtime defenses are too late if the loader already executed malicious logic. The NIST IR 8596 Cyber AI Profile is useful here because it frames AI risk as a lifecycle issue, not just a model-quality issue.

• Require signed artifacts and verify hashes before download and before load.
• Prefer safe serialization formats and disallow arbitrary object deserialization where possible.
• Run model loading in a constrained environment with no ambient secrets and minimal network access.
• Separate training, evaluation, and inference artifacts so a compromised checkpoint cannot automatically reach production.
• Track provenance from source registry to deployment target, including transitive packages and custom operators.

NHIMG case studies such as the JetBrains Marketplace AI Plugin Campaign show how trust in an extension ecosystem can be exploited to reach API keys and workstation context. This is why model supply chain controls must include package review, not just model accuracy checks. These controls tend to break down when teams auto-fetch checkpoints into CI/CD or notebook environments because the loader inherits credentials and network reach by default.

Common Variations and Edge Cases

Tighter artifact controls often increase friction for researchers and platform teams, requiring organisations to balance delivery speed against the risk of executing untrusted model code. That tradeoff becomes sharper in environments that rely on community checkpoints, rapid experimentation, or multi-tenant inference platforms where many teams need flexible access.

There is no universal standard for every model format yet, so policy has to reflect the artifact type and the loader behavior. A PyTorch checkpoint, a containerized inference image, and a hosted model endpoint present different risks. For example, a model stored in a container may be safer than a raw serialized object, but only if the container image, entrypoint, and dependencies are themselves controlled. The 52 NHI Breaches Analysis reinforces the broader lesson: compromise often happens through overlooked trust paths, not obvious application flaws.

One practical edge case is offline model distribution. Some teams assume air-gapped transfer eliminates risk, but it mainly changes the attack path. If a malicious artifact is introduced before scanning or signing, the environment can still be compromised at the first load. Another edge case is “safe” vendor-hosted models that call back to external services, which can create unexpected data egress or dependency risk. Best practice is evolving, but current guidance still favors provenance, sandboxing, and strict egress control over optimistic trust.

Related resources from NHI Mgmt Group

• Why do untrusted AI model files create a larger security risk than many teams expect?
• How can teams reduce risk from AI-generated code in supply chains?
• Why do AI supply chains create identity and access risk?
• Why do AI chatbots create supply chain risk for IAM teams?