AI model files are an attack surface, not just data

At a glance

What this is: This analysis argues that the model file itself is a security boundary, because common formats and companion artefacts can execute code or conceal malicious behaviour.

Why it matters: It matters because AI deployment teams may inherit software-supply-chain risk without applying package-style controls to model artefacts, leaving production systems exposed before the model ever runs.

By the numbers:

• JFrog's security research team found over 100 models containing malicious payloads.
• Over 6,000 models on Hugging Face were affected before they were patched.

Context

AI model files are now part of the software supply chain, but most organisations still review them less rigorously than packages, patches, or container images. That gap matters because a model artefact can be more than data: it may include executable deserialisation logic, unsafe metadata, or companion files that expand the attack surface before any inference happens.

The core governance problem is not whether a model is popular or trusted by the community, but whether the organisation understands what happens when loading it. Public repositories make distribution easy, yet the security model for model files is still immature across many teams. For security leaders, this is a provenance and execution problem, not just an AI operations problem.

Key questions

Q: What should security teams do first when they start loading external AI models?

A: Start by restricting model admission to a small set of approved formats, then force every external model through an isolated review pipeline. That first control matters because the loading step may execute code, process dangerous metadata, or pull in companion files that expand exposure before the model ever serves traffic.

Q: Why do AI model files create security risk before inference even begins?

A: Because the load step can be executable. Some formats deserialise code, some loader libraries interpret metadata, and some conversion workflows add risk outside the weight file itself. The result is that a model can compromise the environment before it produces a single output.

Q: How can teams tell whether model scanning is actually reducing risk?

A: Look for whether scanning is only a pre-admission signal or a true decision gate. If unscanned formats are blocked, provenance is recorded, and exceptions require review, scanning is helping. If teams treat a clean result as proof of safety, the control is performing poorly.

Q: What is the difference between a safer model format and a safe model supply chain?

A: A safer format reduces one technical execution path, but a safe supply chain also covers conversion services, metadata, repository trust, and provenance. In practice, the file format is only one control point. The broader system determines whether the model can be loaded without introducing hidden risk.

Technical breakdown

Why model serialization can become code execution

Some model formats are not passive data containers. Python pickle, for example, can execute arbitrary code during deserialization, which means loading a model can trigger attacker-controlled logic before the application even evaluates the model. Other formats, such as TensorFlow Keras with Lambda layers, can embed custom functions that are also executed at load time. The technical issue is that the loading process itself becomes an interpreter for untrusted content, so the boundary between data and executable code disappears. In practical terms, a model file can act like a malicious package archive rather than a benign asset. See the AI Supply Chain Security and AI-BOM Guide.

Practical implication: Restrict loading paths to formats and libraries that do not execute untrusted code during deserialization.

Why safer formats still leave a supply chain problem

Safetensors removes the most obvious code execution path because it stores only tensor data, but the ecosystem around the file still matters. Conversion services, repository metadata, tokenizer files, and chat templates can all introduce risk even when the weight file itself is safe. That creates a broader trust problem: the file may be clean, yet the workflow that produces or consumes it may not be. For practitioners, the correct unit of analysis is the whole model repository and delivery pipeline, not the extension on a single artefact.

Practical implication: Inspect the full model repository and conversion workflow before promotion, not just the weights file.

How scanning helps but cannot prove model safety

Static scanning tools look for dangerous imports, functions, or known malicious patterns inside model files. That is useful for catching obvious threats, but it behaves like signature-based malware detection: it can miss novel payloads, obfuscated behaviour, or payloads hidden in unusual combinations of otherwise legitimate operations. The limitation is structural. A scanner can reduce exposure, but it cannot certify that a model is harmless because the absence of a known signature is not proof of benign intent. Security teams should treat scanner output as one signal, not a final trust decision.

Practical implication: Use scanning as a gate, then combine it with provenance review, isolation, and controlled promotion.

Threat narrative

Attacker objective: The attacker wants code execution or remote access on the victim system through a model artefact that appeared safe to download.

1. Entry occurs when a user downloads a model file from a public repository or consumer tool and loads it into a production or test environment.
2. Credential or code access occurs when the file's serialization logic or companion artefacts execute attacker-controlled instructions during loading.
3. Impact follows when the attacker gains a foothold, establishes remote access, or plants persistence through the trusted model import path.

Breaches seen in the wild

• Hugging Face Spaces breach 2024: Unauthorised access to Hugging Face Spaces may have exposed secrets users stored for AI apps; tokens were revoked and org tokens removed.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Model artefact governance is now part of identity-adjacent supply chain control: if a downloaded model can execute code on load, then the trust decision is no longer confined to AI teams. It becomes a security governance issue that touches software provenance, execution boundaries, and approval workflows for everything entering production. Practitioners should treat model files like untrusted third-party artefacts rather than inert content.

File safety is not ecosystem safety: a safer weight format reduces one attack path, but repository metadata, conversion services, and loader behaviour can reintroduce risk. That means the real control point is lineage, not format branding. Teams that focus only on the extension of the model file will miss the more important question of who produced it, how it was converted, and what else ships beside it.

AI model provenance should be governed like software provenance: the organisation needs visibility into sources, transformations, and downstream consumers before a model is allowed into a shared environment. The right mental model is an AI-BOM, not a file download. If you cannot explain the model's origin and the artefacts around it, you do not actually know what you are loading.

Scanning is necessary but not sufficient because trust decisions are being made on incomplete evidence: signature checks can flag obvious malicious payloads, but they do not eliminate unknown techniques or risky conversion paths. That creates a false sense of control if the organisation equates “not flagged” with “safe.” The practitioner conclusion is to shift from static detection alone to layered admission control and provenance assurance.

The named failure mode here is unsafe model loading: the organisation assumes the model file is data, while the format, metadata, or loading path can behave like executable code. That assumption collapses the boundary between AI operations and endpoint security. Teams should formalise that collapse as a governance risk, not an isolated technical exception.

Read next: AI Supply Chain Security and AI-BOM Guide

What this signals

AI model provenance will become a baseline control: as model artefacts move into more production environments, teams will need to know origin, conversion history, and accompanying files before approval. That is the practical line between a governed model supply chain and opportunistic downloading.

Format governance will matter more than format preference: the issue is not simply choosing one file type over another, but ensuring that loader behaviour, metadata handling, and repository review are controlled together. Security programmes that ignore the surrounding workflow will keep rediscovering the same exposure in different packages.

AI supply chain risk now extends into the identity of the artefact itself: if a model can behave like executable code, then its trust posture must be treated as part of the broader governance model for AI systems. The next step is to bind model approval to AI supply chain and AI-BOM controls rather than ad hoc downloads.

For practitioners

• Enforce safetensors by default Prefer safetensors for model ingestion and block higher-risk formats in production loading paths unless a risk exception is approved.
• Inspect the full model repository Review weights, configuration files, tokenizer assets, and chat templates together before any model is promoted.
• Treat model artefacts as untrusted code Load external models in isolated environments with restricted network and filesystem access, then promote only after review.
• Track provenance from source to deployment Record where each model came from, who published it, and whether any conversion or repackaging changed the artefact.
• Use scanning as an admission signal, not a guarantee Combine static scanning with manual review and controlled release criteria so a clean scan does not equal automatic trust.

Key takeaways

• Model files can carry executable behaviour, so the load step itself is part of the attack surface, not just the inference runtime.
• Static scanning helps but cannot prove safety, especially when the risky path lives in metadata, conversion, or loader logic.
• Practitioners need provenance controls, isolated promotion, and format restrictions if they want to govern AI model risk effectively.

Related NHI Mgmt Group resources: AI Supply Chain Security and AI-BOM Guide · Attack Surface Management Guide · AI Security Maturity Model · AI Security Platform Buyer's Guide

Key terms

• Model Serialization: Model serialization is the process of packaging a trained model into files that can be stored, shared, and loaded later. In security terms, it matters because the loader may interpret more than data, turning a model file into an execution path if the format allows embedded logic or unsafe metadata.
• Safetensors: Safetensors is a file format for storing machine learning tensors in a way that avoids executing embedded code when a model is loaded. It reduces the risk of malicious deserialisation compared with more flexible formats, but defenders still need provenance checks, integrity validation and trusted sources before importing models.
• Model Provenance: Model provenance is the evidence chain showing where an AI artefact came from, how it was modified, and whether the version in use is the one that was approved. For AI security teams, provenance is the control that turns trust from assumption into verification.
• Static Scanning: Static scanning is the analysis of code, configuration, or files without running them, to find security flaws, policy violations, and exposed secrets early in the development or delivery process. In identity and cloud security, it helps detect hardcoded credentials, weak access controls, misconfigurations, and unsafe dependencies before deployment.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management for practitioners who need stronger control over trusted artefacts and access paths. It gives security and identity teams a common language for governance decisions across modern AI and software supply chains.