Model Scanning should check model artefacts for malicious code, unsafe dependencies, poisoned weights, and suspicious metadata before deployment. Scan every new version, verify provenance, and compare results against approved baselines. Security teams should integrate scanning into release gates, block untrusted artefacts, and investigate changes in behaviour after retraining or import from external sources.
What model scanning needs to catch before deployment
Model scanning is not just a file check, it is a control for deciding whether a model artefact is safe to trust in your build and release path. The practical goal is to catch malicious code, unsafe dependencies, poisoned weights, and suspicious metadata before those artefacts can influence production behaviour or downstream automation.
That means the scan has to look at the whole artefact package, not only the model file itself. In practice, teams should treat every new version, retrained checkpoint, imported third-party model, and regenerated artefact as a fresh trust decision, especially when the source chain is incomplete or the behaviour changes unexpectedly.
Provenance matters as much as content. A useful scanning program compares the incoming artefact against approved baselines, records what changed, and flags unexpected shifts in structure, dependencies, labels, or embedded metadata so reviewers can separate routine model drift from suspicious modification.
How to build scanning into the release gate
Scanning works best when it is part of the release workflow, not a separate after-the-fact review. The release gate should fail closed on untrusted artefacts, require a known source path, and prevent deployment until the scan result is tied to the exact model version that will run.
That also means the control needs strong change tracking. If a model is retrained, fine-tuned, or imported from an external source, the scan should be repeated automatically and the result compared with the previous approved state. Reuse of an old approval is a common mistake because the artefact may look familiar while its dependency chain, weights, or metadata have changed materially.
For teams using shared registries or model hubs, the gate should also separate trusted internal assets from externally obtained artefacts. If the model has crossed trust boundaries, the question is no longer only whether it works, but whether the source, packaging, and surrounding dependencies still match the approved baseline.
What suspicious change signals usually justify escalation
Model scanning becomes most valuable when it is tuned to deviations that matter operationally, not just generic malware signatures. Changes in embedded dependencies, unexpected code paths, altered metadata, unexplained weight differences, or a sudden shift in model behaviour after retraining can all justify deeper review because they suggest tampering, hidden functionality, or an unreviewed supply chain change.
Behavioural change deserves special attention when the model is expected to be stable. If output patterns, routing decisions, or confidence profiles shift without a matching business explanation, treat that as a signal to investigate the upstream artefact and the training or import process rather than assuming the change is benign.
Where model artefacts are assembled from multiple sources, the review should also test whether the final package contains components that were never explicitly approved. This is especially important when teams rely on external code, prebuilt weights, or downloaded artefacts that may have passed through several hands before reaching the release pipeline.
Risk and Threat Considerations
Model scanning reduces the chance that a poisoned or altered artefact enters production through a trusted delivery path. The main risk is that the organisation assumes a model is safe because it came from an approved process, while the artefact itself has been modified, wrapped, or supplied with unsafe dependencies.
Failure mechanism: An attacker or compromised upstream source can embed malicious code, hidden behaviour, or manipulated weights inside an artefact that appears legitimate, then rely on weak provenance checks or skipped rescans to move it into deployment.
Impact: The result can be unsafe model behaviour, compromised release integrity, hidden persistence in the supply chain, and downstream exposure in any application or workflow that trusts the model’s output.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10 addresses the attack and risk surface, while SLSA and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Non-Human Identity Top 10 | NHI-02 — Secret Leakage | Model artefacts may embed credentials or secret material. |
| NHI-03 — Vulnerable Third-Party NHI | Unsafe dependencies in model packages create supply-chain exposure. | |
| Recommendation — Scan artefacts for embedded secrets and block deployment until they are removed. Review third-party components in model artefacts before promoting them. | ||
| SLSA | Supply-chain integrity | Provenance and baseline checks are core to trusted model delivery. |
| Recommendation — Require verifiable provenance for each model artefact before release. | ||
| NIST SP 800-53 Rev 5 | CM-8 — System Component Inventory | Model scanning relies on knowing which artefacts and versions exist. |
| SI-7 — Software, Firmware, and Information Integrity | Scanning and baseline comparison are integrity controls for model files. | |
| Recommendation — Maintain an authoritative inventory of approved model artefacts and versions. Validate model artefacts against approved integrity baselines before deployment. | ||
Practitioner Guidance
What to verify: Confirm that every model artefact is tied to a known source, a recorded baseline, and a scan result for the exact version that is about to ship. If the artefact changed in any material way since the last approval, treat it as a new trust event rather than a routine redeploy.
Decision rule: If the scan cannot explain the provenance of the artefact, or if the model behaves differently after retraining or import without a business explanation, block release and investigate before deployment. Do not accept “known good last time” as evidence for a new build.
What good looks like: The release pipeline rejects untrusted artefacts automatically, scanning runs on every new version, and reviewers can trace each approval back to the model, source, and baseline that were actually assessed.
Practitioner takeaway: The real objective is not to scan models once, it is to make every artefact pass a repeatable trust check whenever its source, content, or behaviour changes.