Model Supply Chain Guide: Malicious Models, Pickle Files and Safetensors

Loading a model file is often the same as running code, and most teams still download model weights from public hubs with less scrutiny than they give an npm package. The default PyTorch format for years was pickle, which can import and call arbitrary Python during deserialisation, and researchers have found malicious models on public hubs that open a remote shell the moment they are loaded. This guide covers the model artefact itself: which file formats can execute code, how safe loading options in PyTorch, Keras and Transformers work and where they stop, what hub scanning catches, and how to build an intake pipeline with pinning, signing and an internal registry. It is for ML platform, application security and supply chain teams. Inventory and AI-BOM practice is covered in the AI Supply Chain and AI-BOM Guide, and behavioural backdoors in weights in the AI Data Poisoning and Backdoors Guide.

Key takeaways

• Pickle is code, not data. Python's own documentation says the pickle module is not secure and that malicious pickle data can execute arbitrary code during unpickling.
• Safetensors removes the execution path for weights. It stores tensors only, so prefer it for every model you download or publish, and convert legacy checkpoints in an isolated job.
• Safe loading flags help but are not a sandbox. PyTorch 2.6 and later default to weights_only=True, and Keras defaults to safe_mode=True, yet both have documented limits and both can be switched off by a single argument.
• Hub scanning is advisory. Hugging Face scans every commit and highlights suspicious pickle imports, but states its scan is not foolproof, and flagged files can still be downloaded.
• Provenance needs pinning and signatures. Pin every production model to a commit hash and a file digest, mirror it into an internal registry, and verify signatures where the publisher provides them.

What sits in a model supply chain

A model you deploy is a bundle of artefacts from several parties, each of which can be tampered with. OWASP's LLM03:2025 Supply Chain entry lists vulnerable pre-trained models, weak model provenance (model cards give no guarantee of origin), malicious LoRA adapters, collaborative model merge and conversion services, and traditional package vulnerabilities as distinct risks.

Classic dependency risk is covered in the Software Supply Chain Security Guide; the rest of this page deals with the model-specific rows.

Why pickle-based model files execute code

Pickle is a stream of opcodes for a small virtual machine, not a passive data layout. Hugging Face's pickle documentation identifies the opcodes that matter: GLOBAL and STACK_GLOBAL import an object by module and name, and REDUCE calls it with arguments. An attacker who controls the file can therefore import a built-in that runs code and call it while the rest of the file still looks like ordinary weights. The same mechanism underlies insecure deserialisation in any language; see the Insecure Deserialisation Guide.

• It was the default: PyTorch checkpoints saved with torch.save (pytorch_model.bin, .pt) are pickle inside a zip, so a large share of older Hub models are pickle.
• It has been abused on public hubs: JFrog reported in February 2024 that it had found around 100 malicious models on Hugging Face, mostly PyTorch and some Keras, including one whose payload opened a reverse shell to a remote host when loaded with torch.load().
• Allowlisting the unpickler is hard: Python documents overriding Unpickler.find_class() to forbid or restrict globals, but warns you must be careful what you allow. Hugging Face notes a custom unpickler alone is not a sufficient fix, citing a CTF bypass.
• Other formats execute code too: Protect AI's Hub scanner looks for Keras Lambda layer exploits as well as pickle, and Keras documents that loading with safe_mode=False can trigger arbitrary code execution.

Model file formats compared

Hugging Face's pickle guidance also suggests loading TensorFlow or Flax checkpoints into PyTorch with from_tf=True or from_flax=True where a repository offers them, and lists MsgPack, Protobuf, Cap'n Proto and Avro as alternative serialisation formats.

Safe loading controls and their limits

PyTorch

• Default since 2.6: torch.load uses weights_only=True when no pickle_module is passed. The restricted unpickler only builds plain tensors and some primitive types needed for state_dicts and cannot import anything dynamically.
• Documented limits: PyTorch states it does not protect against denial of service, that memory corruption is mitigated but may still be possible, and that loaded objects can still cause problems in downstream code.
• Allowlisting: torch.serialization.add_safe_globals() adds globals process-wide, while the safe_globals() context manager limits the allowlist to one block; prefer the context manager. get_unsafe_globals_in_checkpoint() lists non-allowlisted globals by static analysis, and PyTorch notes it may miss types built dynamically.
• Environment overrides: TORCH_FORCE_WEIGHTS_ONLY_LOAD=1 forces every call to weights_only=True; set it in serving images. TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 does the opposite for calls that do not pass the argument; alert if it appears in any image or job definition.
• The escape hatch: any code calling torch.load(..., weights_only=False) restores full pickle behaviour, and PyTorch's own error message warns this can lead to arbitrary code execution. Make it a code review and static analysis finding.

Keras, Transformers and plain Python

• Keras: keras.saving.load_model(..., safe_mode=True) is the default and disallows unsafe lambda deserialisation for .keras files. Search code for safe_mode=False.
• Transformers: trust_remote_code=True loads Python from the model repository. Treat each use as a third-party code dependency: review the files, pin the repository to a commit, and record the exception.
• Plain pickle and joblib: scikit-learn and other classic ML models are often pickled. Python's advice is never to unpickle data that could have come from an untrusted source or been tampered with, and to consider hmac signing where you control both ends.

Choosing between controls

• You control the publisher: publish safetensors only and sign it; consumers then need no unpickling at all.
• Third-party model only in pickle: convert once in an isolated, network-less job, scan the result, publish the safetensors copy internally, and never load the original in production.
• Old checkpoints that store a whole nn.Module: PyTorch recommends weights_only=False for these, so they should only ever come from your own trusted training runs. Re-save them as a state_dict to retire the exception.
• Small teams: the cheapest effective control is a rule that production loads only safetensors from an internal mirror. Large organisations add an intake service with scanning, signing and an approval workflow.

Scanning and its limits

Scanning is a useful tripwire and a poor guarantee. Static scanners read the pickle opcode stream without executing it and flag imports that look dangerous; they can be evaded by imports not on the list, and they say nothing about behavioural backdoors in weights.

Provenance, signing and pinning

• Pin to a commit, not a name: loading org/model without a revision follows the repository's default branch, so a compromised publisher token changes what you load. Pin a commit hash in code and in the AI-BOM.
• Record file digests: store SHA-256 digests of each approved file and verify them at load time; D3FEND's File Hashing technique describes the control.
• Verify signatures where available: the OpenSSF-associated model_signing project (sigstore/model-transparency) signs a model directory with Sigstore, keys, certificates or PKCS #11 hardware, produces a Sigstore bundle holding an in-toto statement of file digests, and on verify recomputes the hashes. With Sigstore, signing events land in a transparency log publishers can monitor.
• Signed commits are not signed models: Hugging Face notes that GPG-signed commits guarantee origin, not safety.
• Protect your own publishing identity: publish from CI with short-lived Trusted Publisher tokens rather than a stored write token, so a leaked CI secret cannot replace your models. See the Hugging Face Security Guide and the AI Model Provenance and Watermarking Guide.

Building a model intake pipeline

1. Inventory what you load today: grep code and notebooks for from_pretrained, torch.load, pickle.load, joblib.load, load_model and trust_remote_code; record source, revision and format in the AI-BOM.
2. Set the format policy: safetensors and Keras v3 with safe_mode=True allowed; pickle formats and remote code only by documented exception.
3. Stand up an internal mirror: a model registry or artefact store that production can reach, with production egress to public hubs blocked.
4. Intake job: download into an isolated runner with no credentials and no network after download, check format, run static scans, convert pickle to safetensors, compute digests, sign, and publish to the mirror.
5. Audit mode first: for two to four weeks log every load that would be blocked (wrong format, unpinned revision, unsigned file) without blocking, and fix the top offenders.
6. Enforce in rings: set TORCH_FORCE_WEIGHTS_ONLY_LOAD=1 and mirror-only loading in staging, then one production service, then all. Rollback is reverting the image variable or the egress rule for the affected service, with an exception ticket.
7. Continuous checks: re-scan mirrored models when scanner rules update, and review exceptions quarterly.

Detection and measurement

• Load-time logging: log model identifier, revision, file digest and loader arguments for every model load in serving and training jobs.
• Runtime signals: alert on child processes (shells, interpreters) or new outbound connections from model-serving and notebook processes during load, which maps to ATT&CK T1059.006 Python and ATLAS AML.T0011.000 Unsafe AI Artifacts.
• Code and config signals: CI rules that fail on weights_only=False, safe_mode=False, unreviewed trust_remote_code=True and TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD.

Metrics:

• Percentage of production model loads from the internal mirror.
• Percentage of production models in safetensors or another code-free format.
• Percentage of model references pinned to a commit hash with a recorded digest.
• Number of open trust_remote_code and pickle exceptions, and their age.
• Median time from a scanner alert on a mirrored model to quarantine.

Framework mapping

Common failure patterns

• Trusting the hub badge: a model without a warning is treated as safe, although Hugging Face says its scan is not foolproof and JFrog showed flagged models could still be downloaded.
• Silencing the error: a developer hits a weights_only unpickling error after a PyTorch upgrade and adds weights_only=False, reopening the code execution path for every checkpoint that code loads.
• Remote code left on: trust_remote_code=True copied from a model card into production, unpinned, so the next commit to that repository runs in your serving environment.
• Loading by name in production: no revision pin and no digest, so a publisher account takeover becomes your incident.
• Notebooks as the soft spot: data scientists load experimental models on laptops or notebook servers that hold cloud credentials and Hub write tokens. Isolate experimentation from credentials. See the AI Coding Agents Security Guide for the same pattern with agents.
• Converting in place: running the pickle to safetensors conversion on a build host with secrets, which executes the very payload you were trying to remove.

Practitioner checklist

• Inventory every model load and record source, revision, format and digest in the AI-BOM.
• Make safetensors the default format for models you publish and consume.
• Set TORCH_FORCE_WEIGHTS_ONLY_LOAD=1 in serving images and fail CI on weights_only=False and safe_mode=False.
• Disallow trust_remote_code except reviewed, pinned exceptions.
• Serve production only from an internal model mirror and block egress to public hubs.
• Run intake and conversion in an isolated runner with no credentials and no network.
• Verify digests at load and signatures where publishers sign with model_signing.
• Monitor model-serving processes for child processes and unexpected outbound connections.
• Publish your own models from CI with short-lived tokens, never a stored write token.

Standards and references

• OWASP: Top 10 for LLM Applications 2025 (LLM03 Supply Chain)
• MITRE ATLAS: AI Supply Chain Compromise
• Python Software Foundation: pickle, Python object serialization
• PyTorch: Serialization semantics
• Hugging Face: Safetensors
• Hugging Face: Pickle Scanning
• Sigstore: Model Transparency (model_signing)
• JFrog: Data Scientists Targeted by Malicious Hugging Face ML Models with Silent Backdoor

Related NHI Mgmt Group resources: AI Supply Chain and AI-BOM Guide · Insecure Deserialisation Guide · Open-Weight Model Security Guide · Hugging Face Security Guide · SLSA and Build Provenance Guide