How do attackers use Model Serialisation Attack?

How attackers weaponise model serialisation

model serialisation attack target the trust boundary between a saved model artefact and the code that loads it. The attacker’s goal is usually not to break the model itself, but to abuse the loader, conversion pipeline, or deserialiser so that malicious payloads execute, object state is altered, or the resulting model behaves in a way the defender did not intend.

That makes the attack effective anywhere models move between training, packaging, storage, distribution, and deployment. In practice, the dangerous moment is often loading, not training, because many teams treat model files as data rather than executable input.

When the serialised object format supports rich object graphs, custom classes, or embedded callbacks, an attacker can hide logic inside the artefact. The result may be remote code execution, tampering with inference results, or a model that appears valid but has corrupted weights, metadata, or execution hooks.

Where the attack usually enters the pipeline

Attackers generally look for weak points in model sharing and promotion workflows: downloaded checkpoints, third-party model hubs, CI artefacts, experiment exports, and conversion jobs that move models across frameworks. The safer the organisation thinks the path is, the more likely the artefact will be handled with excessive trust.

One common pattern is social engineering around “just load this file and verify it.” Another is abusing conversion utilities, format bridges, or notebooks that automatically inspect artefacts. A malicious file can exploit unsafe parsing long before the model is ever used for inference.

Defenders should assume that any unverified serialised model is potentially executable content. The practical control is to treat model artefacts as untrusted inputs until they pass integrity checks, provenance review, and security review inside a controlled environment.

What defenders should verify before loading a model

The first question is whether the file came from a trusted source and whether its format is appropriate for the environment that will open it. If a team cannot explain the artefact’s origin, signature, and expected format, it should not be loaded in a privileged runtime.

Trusted formats and explicit validation reduce the chance that deserialisation will interpret attacker-controlled structure as code. A strong practice is to separate “inspect” from “execute”, so that potentially dangerous artefacts are analysed in a low-privilege sandbox before any production system touches them.

For organisations that publish, consume, or transform machine-learning artefacts, supply-chain hygiene matters as much as application hygiene. Guidance from the NIST Cybersecurity Framework 2.0 and the CISA cyber threat advisories reinforces the need to identify where untrusted content enters the environment and to control those ingestion paths deliberately.

Risk and Threat Considerations

Model serialisation attacks are attractive because they turn a routine operational action, loading a model, into a compromise point. If the loader trusts the artefact too much, an attacker can move from file delivery to code execution, persistence, or model tampering without needing a separate exploit chain.

Failure mechanism: Unsafe deserialisation, parser abuse, or format conversion logic interprets attacker-controlled fields as executable behaviour or corrupts model state.

Impact: The result can be arbitrary code execution, poisoned inference, denial of service, or silent compromise of downstream systems that rely on the model’s output.

Practitioner Guidance

What to verify: Confirm the artefact’s source, integrity, expected format, and signature before it reaches any environment with production credentials or network reach. If any of those checks are missing, treat the file as hostile until proven otherwise.

What not to automate: Do not allow automated jobs to pull and load arbitrary models from external repositories into privileged runtimes without an approval gate. The convenience of frictionless model promotion is exactly what attackers rely on.

Practitioner takeaway: The key judgement is to treat model files as potentially executable supply-chain inputs, not as inert data, and to move all inspection and validation ahead of any privileged load step.

Related resources from NHI Mgmt Group

• Why do attackers often check model availability before trying to generate content?
• How do attackers turn a supply-chain incident into wider NHI compromise?
• What does AI model abuse reveal about the current NHI threat surface?
• How do attackers operationalise stolen OAuth tokens at scale?