An AI model that has been altered or packaged to harm users, systems, or data. The danger may come from embedded code, hidden instructions, poisoned weights, or deceptive packaging that causes credential theft, data leakage, or unauthorized system behavior after download or execution.
Expanded Definition
A malicious model is not simply an inaccurate or low-quality model. It is a model artefact that has been intentionally tampered with, wrapped, or distributed to produce harm when another party loads, runs, fine-tunes, or trusts it. The harmful element may live in the weights, the surrounding code, the model package, or the instructions and metadata that shape downstream behaviour. That boundary matters because the risk is not limited to the model’s predictions; it includes everything the model can reach once it is integrated into an application, pipeline, or agentic workflow.
In practice, the term covers multiple attack surfaces: poisoned checkpoints, backdoored models, malicious wrappers, and deceptive distribution packages. It does not describe ordinary model error, bias, or unsafe prompting alone. Guidance is still emerging on how to classify some packaging and supply-chain variants, but the security consensus is clear that trust must extend beyond the model file itself to provenance, integrity, and execution context. The OWASP Non-Human Identity Top 10 is relevant here because a model that can invoke tools or operate inside a system often inherits machine-like access and therefore needs explicit trust controls.
A common misunderstanding is to treat model provenance as a documentation problem. For malicious models, provenance is a control problem: who signed it, who built it, what dependencies it carries, and what it can do after import.
Examples and Use Cases
Malicious models appear in several real-world patterns that practitioners should recognise:
- A public checkpoint is downloaded because it performs well in benchmarks, but it contains hidden behaviour that activates on a specific trigger phrase or input pattern.
- A model package includes wrapper code that phones home, pulls remote instructions, or exfiltrates environment data when the model is loaded.
- A fine-tuned model inherits poisoned behaviour from a compromised training source, making the backdoor hard to spot in ordinary evaluation.
- A supposedly helpful assistant model is bundled with integration code that abuses local tool access or API credentials after deployment.
- A third-party model artifact is republished under a trusted name, creating deceptive packaging that defeats informal review and casual reuse.
The tradeoff is familiar to teams that move quickly: broader model reuse improves delivery speed, but it also widens the trust boundary. A model that is easy to import is also easy to misplace into production without sufficient inspection, especially when teams optimise for capability rather than provenance.
For distribution and artifact trust, the model supply chain should be treated with the same care as other software dependencies, because the payload can be code-like even when it looks like data.
Security Implications
When a malicious model is trusted, the failure is usually not limited to an incorrect answer. The model can become a delivery mechanism for credential theft, data leakage, unsafe tool execution, or policy bypass inside the host environment. That makes the blast radius larger than a normal model-quality issue because the artefact may carry both deceptive content and executable influence over downstream systems.
The most important failure mechanism is trust inversion. Teams assume a model is passive data, then place it into a pipeline or agent with access to files, prompts, tokens, plugins, or internal services. If the model package contains hidden instructions or embedded logic, the loader or runtime may execute behaviour that the defender never reviewed. Poisoned weights can also encode targeted backdoors that remain invisible in general testing, only surfacing under narrow conditions. The result is a control gap between model evaluation and real operational behaviour.
Practitioners often miss that malicious-model risk is amplified by reuse at scale. One compromised artifact can propagate across many projects, making provenance failures systemic rather than local. In NHIMG terms, that is why machine-like execution trust must be explicit whenever a model can act beyond static prediction.
Domain and Governance Relevance
In AI security, the term sits at the intersection of model integrity, supply-chain assurance, and runtime containment. It matters most when organisations accept external models, redistribute fine-tuned models, or let models participate in tool use, retrieval, or orchestration. In those settings, the central governance question is not whether the model appears useful, but whether its origin, contents, and permitted actions are verifiable.
For identity and access governance, the relevance becomes material when a model is embedded in an autonomous workflow or agentic system. At that point, the model is no longer just analytical output; it is an actor with potential access paths. That changes how provenance, approval, and revocation should work, because a compromised model can inherit the authority of the environment that loads it.
For NHI practice, the key shift is to treat the model artifact as part of the trust chain for non-human execution. If it can invoke tools, reach services, or influence transactions, then its lifecycle must be governed alongside the credentials and permissions that make those actions possible.
Risk and Threat Considerations
Malicious models create a supply-chain and execution risk because the harmful logic may be hidden inside the model artifact, surrounding package, or deployment wrapper. The threat is especially serious when teams source models from unvetted repositories or integrate them into systems that expose secrets, internal data, or tools.
Failure mechanism: Defenders assume the model is inert data, but the artifact contains poisoned weights, hidden triggers, or embedded code that activates after load, fine-tune, or tool invocation. That can produce credential access, exfiltration, or unauthorised actions while appearing normal in standard tests.
Impact: The organisation can lose confidentiality, integrity, and control at the same time. A compromised model may leak prompts or data, call internal services inappropriately, or propagate the same malicious behaviour to every downstream environment that reuses it.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10 and MITRE ATT&CK address the attack and risk surface, while CIS Controls v8, NIST CSF 2.0 and NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Non-Human Identity Top 10 | NHI-01 — Asset Inventory and Ownership | Malicious models behave like non-human artifacts that need owned trust and provenance. |
| Recommendation — Inventory model artifacts and assign ownership before allowing them to influence systems. | ||
| CIS Controls v8 | 15 — Service Provider Management | Third-party model supply chains create supplier trust and verification needs. |
| Recommendation — Vet external model sources and require integrity checks before deployment. | ||
| MITRE ATT&CK | T1027 — Obfuscated Files or Information | Deceptive packaging and hidden payloads rely on concealment to evade review. |
| Recommendation — Inspect model packages for concealment patterns and unexpected embedded content. | ||
| NIST CSF 2.0 | PR.DS — Data Security | Model artifacts can expose data, secrets, or sensitive prompts when mishandled. |
| Recommendation — Protect model inputs, outputs, and artifacts with controls that prevent leakage. | ||
| NIST AI RMF | MAP — Map | Model provenance and intended use must be mapped before trusting the artifact. |
| Recommendation — Document model origin, purpose, and dependencies before operational use. | ||
Practitioner Guidance
Why practitioners should care: Malicious models are a trust-boundary problem, not just a model-quality problem. If your workflow accepts external artifacts, the security decision is whether the model is allowed to influence execution before its provenance and behaviour are sufficiently trusted.
Common misunderstanding: Teams often review the model’s outputs but not the artifact itself. That misses the possibility of embedded payloads, deceptive packaging, and hidden behaviour that only appears after import or integration.
Practitioner takeaway: Treat model intake as an approval point, not a download event, whenever the artifact can affect systems, data, or non-human execution.
Related resources from NHI Mgmt Group
- How can organisations reduce the risk of malicious model supply chain attacks?
- What does a fully malicious server threat model reveal in a password manager audit?
- Why do open AI model ecosystems increase the risk of secrets exposure and malicious code execution?
- Why do malicious commits that target developer tools create a different risk model than classic package supply chain attacks?
