Data and model poisoning is the insertion of malicious, biased, or misleading information into training data, fine tuning inputs, or model artifacts. The goal is to alter how a model behaves at inference time. In security terms, it can create hidden backdoors, unsafe recommendations, or systematic misinformation.
Expanded Definition
Data and model poisoning is a corruption problem in the lifecycle of machine learning systems. It covers malicious or misleading training examples, poisoned fine-tuning corpora, tampered labels, and altered model artefacts that change how a model behaves after deployment. The core issue is not just bad data quality. It is intentional influence over model behaviour, often in ways that are difficult to detect during normal testing.
The term sits between data integrity, model supply-chain integrity, and adversarial ML. It excludes ordinary dataset noise unless the contamination is deliberate or sufficiently targeted to affect model outputs. Guidance across the field is still evolving, but the broad security consensus is that poisoning can create persistent behavioural changes that survive retraining cycles, evaluation blind spots, or downstream reuse. For practical orientation, the OWASP Non-Human Identity Top 10 is relevant when poisoned pipelines depend on service accounts, tokens, or automated ingestion paths that silently expand trust.
Examples and Use Cases
Poisoning appears anywhere model behaviour is shaped by external or semi-trusted inputs. The exact form depends on where the attacker can influence the learning or tuning process.
- A public dataset used for pretraining is seeded with subtle mislabeled samples so the model learns an incorrect association that later appears in inference.
- A fine-tuning corpus is altered to bias responses toward a preferred outcome, such as ranking one vendor, policy, or class of content above others.
- Model artefacts are replaced or modified in a build pipeline so a deployed model behaves normally in most cases but fails predictably on a trigger pattern.
- A retrieval or feedback loop ingests low-quality or adversarial user inputs, gradually shifting the model toward unsafe or misleading outputs.
- An automated ingestion job accepts poisoned content through a trusted connector, showing how weak pipeline trust can become an attack surface rather than a simple data hygiene issue.
The tradeoff is that ML systems often need broad, open, or continuously refreshed data sources to stay useful. That same openness increases the chance that untrusted content, if not well governed, can shape model behaviour in ways operators do not notice until after deployment.
Security Implications
When poisoning succeeds, the model may appear healthy while carrying a hidden behavioural defect. That makes this issue especially difficult because ordinary accuracy checks can miss targeted backdoors, rare trigger conditions, or class-specific bias. The result can be misleading recommendations, distorted detections, unsafe automation, or systematic decision errors that are baked into the model rather than introduced at query time.
Poisoning also creates governance and incident-response problems. Teams may investigate the wrong layer if they assume the model itself is faulty, when the real problem is compromised source data, ingestion integrity, or an altered training artefact. A common practitioner reality is that the more automated the training pipeline, the easier it is for poisoned content to move from source to model without human review. In that sense, the failure is often one of trust boundary design, not just model robustness.
Domain and Governance Relevance
From an AI security perspective, data and model poisoning is a lifecycle integrity issue: who can contribute data, who can approve model artefacts, and what evidence proves the training set or checkpoint has not been manipulated. Governance therefore needs to cover provenance, change control, and validation of both inputs and outputs, not just final model performance.
The term also has a material identity and access dimension where automated pipelines, service accounts, and machine-to-machine uploads are involved. If those non-human identities are over-privileged or poorly inventoried, poisoning becomes easier to introduce and harder to trace. For that reason, model governance and NHI governance intersect when the training path depends on trusted automation. The key question is not whether an identity exists, but whether it can alter the data or artefact lineage without detection.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATLAS address the attack surface, NIST AI RMF, NIST AI 600-1 and CIS Controls v8 set the technical controls, and ISO/IEC 42001:2023 define the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | MAP — AI Risk Management Process | Poisoning is an AI lifecycle risk that distorts model trustworthiness. |
| Recommendation — Map data and model poisoning risks into your AI risk process and validate training-data provenance. | ||
| NIST AI 600-1 | DATA — Data Governance and Integrity | The term directly concerns tampered training data and artefacts. |
| Recommendation — Apply data integrity controls to training sets, fine-tunes, and model artefacts before release. | ||
| MITRE ATLAS | AML.TA0002 — Poisoning | ATLAS explicitly covers poisoning as an adversarial AI technique. |
| Recommendation — Use ATLAS poisoning techniques to hunt for manipulated inputs and model behaviour shifts. | ||
| ISO/IEC 42001:2023 | 8.3 — AI Risk Treatment | Organisation-level AI governance must address poisoned data and artefact handling. |
| Recommendation — Embed poisoning scenarios into AI risk treatment and governance reviews. | ||
| CIS Controls v8 | 3 — Data Protection | The subject depends on protecting the integrity of data used by systems. |
| Recommendation — Restrict and monitor access to training data and model artefact repositories. | ||
Related resources from NHI Mgmt Group
- Why does AI data poisoning create governance risk beyond model accuracy?
- What is the difference between data poisoning and a backdoor in a machine learning model?
- What are the signs that a regression model has been affected by training data poisoning?
- Who is accountable when sensitive data is sent to an AI model from the browser?
