Every model, adapter and dataset your teams pull from a public hub is third-party code and data entering production, and most organisations still accept them with less checking than a software library. A pickle-based checkpoint can run code the moment it is loaded, a fine-tuning adapter can alter a trusted base model, and a model card says nothing reliable about who really built the file. Tools that scan model files, generate AI bills of materials (AI-BOMs), verify signatures and check licences are now sold as standalone products and as features of wider AI security platforms. This vendor-neutral buyer's guide is for ML platform, application security and software supply chain teams choosing such tools. The mechanics of unsafe formats and safe loading are in the Model Supply Chain Guide; runtime guardrails and AI posture tools are in the AI Security Platform Buyer's Guide.
Key takeaways
- Scanning is a tripwire, not a guarantee. Hugging Face states its own pickle import scan is not 100% foolproof, and static scanners say nothing about behavioural backdoors in weights. Buy scanning as one gate among several.
- Gate where models enter, not only where they are stored. The most useful tools sit in the intake path: a proxy or mirror in front of public hubs, a CI step, and a registry admission check.
- Provenance needs signatures and pinning. OWASP's LLM03:2025 entry notes that model cards give no guarantee of origin. Prefer tools that verify signatures (for example the OpenSSF model signing format) and pin commits and file digests.
- The AI-BOM is the inventory you will be asked for. Choose tools that emit a standard format such as CycloneDX ML-BOM so the data flows into your existing SBOM and vulnerability tooling.
- Licences are a supply chain risk too. Model and dataset licences can restrict commercial use or redistribution; automated checks prevent a legal problem becoming a recall.
Define requirements
- Sources: public hubs, vendor model catalogues in your cloud, internal training runs, partner deliveries and models embedded in purchased software.
- Artefact types: pickle checkpoints (.bin, .pt, .pth, .ckpt, .pkl), safetensors, Keras (.keras, .h5), ONNX, GGUF, LoRA and PEFT adapters, tokenisers, modelling code loaded with trust_remote_code, notebooks and datasets.
- Volume and size: number of models and versions per month and the largest file size you must scan; multi-gigabyte weights change scanning cost and latency.
- Enforcement points: developer workstation, CI pipeline, artefact proxy or mirror, model registry admission, and deployment gate.
- Policy outcomes: what blocks (unsafe imports, unsigned model from a sensitive source, prohibited licence) and what warns.
- Outputs: AI-BOM format, findings into your vulnerability management or ticketing tool, and audit evidence for regulators and customers.
- Data handling: whether proprietary models must be scanned on premises or in your own cloud account rather than uploaded to a supplier.
Capability areas
| Capability | What it does | Limits to probe |
|---|---|---|
| Model file scanning | Parses serialised files without loading them and flags code execution paths, such as dangerous pickle imports or Keras Lambda layers | Deny-list evasion; formats supported; files too large to scan; nested archives |
| Modelling code and notebook analysis | Reviews Python shipped with a model repository and flags network calls, shell execution and obfuscation | Depth of analysis; handling of changes between commits |
| Hub proxy or mirror | Fronts public hubs so models are scanned, approved and cached internally before use | Bypass by direct downloads; support for your registries and clouds |
| Signature and provenance verification | Verifies model signatures, publisher identity and file digests; records commit hashes | Support for OpenSSF model signing bundles, Sigstore keyless and certificate signing; policy on which identities to trust |
| AI-BOM generation | Inventories models, datasets, adapters, frameworks and dependencies | Standard output (CycloneDX ML-BOM, also standardised as ECMA-424, or SPDX 3.0 AI and Dataset profiles); accuracy on real repositories |
| Licence and terms checks | Reads model and dataset licences and flags restricted use | Custom licences and acceptable use policies; dataset licence lineage |
| Dataset checks | Scans datasets for secrets, personal data, malicious files and provenance gaps | Scale; formats; overlap with your data security tools |
| Behavioural or backdoor testing | Runs models in a sandbox or against test prompts to look for hidden triggers or unexpected network activity | Coverage is partial by nature; ask what it cannot detect |
Where identity fits
Model intake depends on credentials: hub access tokens with write scope, registry service accounts, CI workload identities that sign models, and cloud roles that read training buckets. Leaked hub tokens have exposed organisations' private models before; see Hugging Face API Tokens Exposed 2023. Ask whether a tool detects hub tokens in code and datasets, signs with workload identity rather than personal keys, and runs with read-only, scoped service accounts. See the Hugging Face Security Guide and the Model Signing and OpenSSF Model Signing Guide.
Evaluation criteria
Weights are illustrative; an organisation that only consumes hosted APIs needs far less than one fine-tuning open-weight models.
| Criterion | Weight | Evidence to request |
|---|---|---|
| Detection on your test corpus | 20% | Results on benign models plus safe, non-functional test files that exercise known unsafe patterns |
| Format and source coverage | 15% | Written list of formats, maximum file size, hubs, registries and clouds supported |
| Enforcement integration | 15% | CI plugins, proxy or mirror mode, registry admission hooks, policy as code |
| Provenance and signing | 15% | Verification of model signing bundles, digest pinning, identity policy |
| AI-BOM quality and format | 10% | Sample CycloneDX or SPDX output from one of your repositories |
| Licence analysis | 10% | Correct classification of a set of model and dataset licences you choose |
| Deployment and data handling | 10% | Self-hosted or in-account option; what leaves your environment; retention |
| Explainability and false positives | 5% | Clear reason per finding; suppression with expiry and approval |
Questions to ask vendors
- Which serialisation formats do you parse natively, and what happens to a format or file size you do not support: block, warn or pass?
- Is your pickle analysis an allow list or a deny list, and how do you handle imports you have never seen?
- Do you ever load or execute a model during scanning, and if so, in what isolation?
- Can you verify OpenSSF model signing bundles and enforce which signing identities we trust per source?
- Which AI-BOM formats and versions do you export, and can we ingest them into our existing SBOM platform?
- How do you detect malicious or altered LoRA adapters and modelling code, not only weight files?
- How do you prevent developers bypassing the gate by downloading directly from a hub?
- Can scanning run entirely in our environment for proprietary models? What telemetry do you collect?
- What can your product not detect, in writing?
Red flags
- "Detects all malicious models": no scanner can prove a model is safe, and behavioural backdoors evade file scanning.
- Pickle only: no coverage of Keras, adapters, modelling code or datasets.
- Proprietary inventory format only: no CycloneDX or SPDX export.
- Upload required: proprietary models must leave your environment to be scanned.
- Advisory only: findings in a dashboard but no way to block intake or deployment.
- No signature support: provenance reduced to a hub badge or a model card.
Proof of concept
- Build a test corpus. Include 20 to 30 popular benign models in several formats, adapters, a repository with custom modelling code, a dataset, and safe test files supplied by your own red team that trigger unsafe-pattern detection without a working payload. Handle them only in an isolated environment.
- Measure detection and noise. Record true positives, false positives and unsupported files per tool.
- Test enforcement. Put the tool in front of a hub mirror or CI job and confirm that a flagged model is blocked, an approved model flows through, and a direct download is detected or prevented.
- Test provenance. Sign a model in your pipeline, alter one file, and confirm verification fails; check identity pinning rules.
- Export an AI-BOM. Import it into your SBOM or asset tooling and check accuracy against what you know is in the repository.
- Check licences. Run a set of models with permissive, restricted and custom licences and compare the results with your legal team's view.
- Score, decide and plan rollout. Start in warn mode on intake, then block for production sources once false positives are understood.
How NHI Mgmt Group can help
We provide independent advice on AI supply chain security, model provenance and the identities that AI pipelines depend on, including evaluation support. Browse vendors in our products directory or contact us.
Related NHI Mgmt Group resources: Model Supply Chain Guide · AI Supply Chain and AI-BOM Guide · Model Signing and OpenSSF Model Signing Guide · Open-Weight Model Security Guide