Poisoned Model

Expanded Definition

A poisoned model is not just an inaccurate model. It is an artefact whose learned weights, embedded logic, or surrounding package content have been manipulated so that behaviour appears legitimate while hiding malicious intent, unsafe execution paths, or covert backdoors. In practice, the term is used most often in shared, downloaded, or reused model environments where trust is placed in the artefact before it is fully verified.

The boundary to keep clear is that model poisoning is about the model artefact itself, while prompt injection and runtime abuse affect the model after deployment. A poisoned model can still answer normally for most requests, which is why the issue is easy to miss during casual testing. Guidance from the OWASP Non-Human Identity Top 10 is relevant when the model is distributed, retrieved, or authorised through machine-facing trust relationships that need explicit governance.

For practitioners, the common implementation reality is that model trust is often inherited from source reputation rather than verified from artefact integrity, provenance, or behaviour under adversarial probing. That makes the term as much about supply chain trust as about model quality.

Examples and Use Cases

Poisoned models show up in a few recurring places across AI development and deployment workflows:

• A team downloads a community model and fine-tunes it, unaware that the base model contains a hidden response trigger.
• A model package includes extra files or scripts that execute during loading, turning model adoption into an execution risk.
• A shared internal model is modified upstream so that it behaves normally in tests but produces unsafe output for a narrow trigger phrase.
• An ML platform caches and redistributes a model artifact without validating provenance, allowing a tainted version to spread across projects.
• An agentic application loads a compromised model and then treats its outputs as trusted instructions for downstream tools.

The main trade-off is convenience versus assurance. Reusing external models accelerates delivery, but each additional trust hop increases the chance that malicious behaviour is inherited rather than created internally.

Security Implications

The security problem with a poisoned model is that compromise can arrive before the model is even used in production. If hidden behaviour is embedded in the artefact, ordinary validation may miss it because the model still performs well on standard evaluation sets. That creates a dangerous confidence gap: defenders may believe they are assessing a benign asset while actually importing an attack surface.

Consequences can include covert exfiltration through model outputs, unsafe tool invocation in agentic workflows, policy bypass, manipulated classification decisions, or the spread of compromised artefacts across downstream consumers. In a shared environment, one poisoned model can create repeated exposure wherever it is cloned, fine-tuned, or embedded into products.

A practical warning sign is inconsistency between expected performance and behaviour under narrow triggers, unusual package contents, or unexplained changes in output style after model ingestion. The risk is amplified when model provenance, signing, and review are weak or absent.

Domain and Governance Relevance

Poisoned models sit at the intersection of AI security, software supply chain assurance, and machine-facing trust. In NHI and agentic AI environments, the issue becomes more serious because the model is often not just a prediction engine. It may also influence tool use, workflow decisions, or delegated action.

That shifts governance from "is the model accurate?" to "is the model trustworthy enough to be granted execution-adjacent influence?" Organisations need clear ownership for model intake, provenance checks, approval rules, and downstream dependency tracking. When a model is reused by multiple teams, the governance problem becomes systemic because one compromised artefact can contaminate many products or automations.

The key domain lesson is that model quality and model trust are not the same control objective. A poisoned model can be high performing and still be unsafe to deploy.

Risk and Threat Considerations

Poisoned models create supply chain and trust-breach risk because the compromise is embedded in a reusable artefact rather than introduced only at runtime. That makes the exposure durable, portable, and hard to detect through ordinary functional testing.

Failure mechanism: An attacker or malicious contributor inserts backdoored behaviour, unsafe code paths, or trigger-based responses into the model or its package so that the artefact behaves normally except under specific conditions. The same mechanism can survive redistribution, fine-tuning, and repeated reuse.

Impact: Compromised models can produce unsafe outputs, trigger unauthorized tool actions, contaminate downstream applications, and undermine confidence in training and deployment pipelines.

Practitioner Guidance

Why practitioners should care: The main governance mistake is treating model sourcing as a procurement convenience instead of a security decision. For poisoned models, the material question is whether the artefact can be trusted before it is allowed to shape outputs or actions.

Common misunderstanding: Good benchmark performance does not rule out hidden malicious behaviour. A model can be accurate on normal prompts and still contain narrow trigger logic or compromised package content.

Practitioner takeaway: Treat model provenance, integrity, and reuse rights as part of the security approval process, not as an afterthought once the model is already embedded in production workflows.

Related resources from NHI Mgmt Group

• Why do poisoned chat templates matter if the model weights are unchanged?
• What breaks when training data is poisoned before model deployment?
• How can teams tell whether an AI model has been poisoned or influenced?
• Why can a tiny amount of poisoned data still create major model risk?