Poisoned training data changes what a model learns, and the change stays in the model after the bad data is gone. MITRE ATLAS records this as AML.T0020 Training Data Poisoning, placed under the Persistence tactic (AML.TA0006) in its 2026.10 data: adversaries add, remove or modify samples, alter labels or manipulate feedback and data collection to cause targeted errors, unsafe behaviour, degraded performance or backdoors triggered by specific inputs. This guide treats AML.T0020 as an ATLAS technique to detect and test for: where poisoning enters a training or fine-tuning pipeline, what the ATLAS cases teach, which ATLAS mitigations apply and how to test for backdoors. The general theory of poisoning and backdoors is in the AI Data Poisoning and Backdoors Guide, which this page links rather than repeats. It is for ML platform, data engineering and AI security teams.
Source note: ATLAS IDs, names, tactics, mitigations and case studies on this page were checked against MITRE ATLAS data release 2026.10 (mitre-atlas/atlas-data on GitHub). Confirm against the official text before relying on it.
Key takeaways
- Persistence, not a one-off. ATLAS places AML.T0020 under Persistence because the behaviour survives in the trained model even after the poisoned data is no longer accessible, so deleting the bad records does not fix a model already trained on them.
- Two routes in. ATLAS names direct poisoning after access to a training pipeline, and poisoned datasets introduced through AI Supply Chain Compromise (AML.T0010).
- Small and cheap can be enough. Carlini and co-authors showed that 0.01% of LAION-400M or COYO-700M could have been poisoned for about 60 US dollars by buying expired domains.
- Five ATLAS mitigations do most of the work. AML.M0025 Maintain AI Dataset Provenance, AML.M0007 Sanitize Training Data, AML.M0008 Validate AI Model, AML.M0023 AI Bill of Materials and AML.M0005 Control Access to AI Models and Data at Rest.
- Test with planted poison. AML.M0035 AI Red Team asks you to introduce controlled poisoned records or triggers into representative pipelines and check that provenance, sanitisation, validation and rollback controls catch them.
What the technique covers
ATLAS describes AML.T0020 for predictive, generative and agentic AI, at maturity "Realized". The attacker's actions fall into four groups, each with a different control point.
| Action (ATLAS wording) | Example in an enterprise pipeline | Primary control point |
|---|---|---|
| Add data samples | Injected records in a fine-tuning set; malicious pages in a web crawl | Source allow-lists, provenance, deduplication and outlier filtering |
| Remove or modify data samples | Deleting edge cases so a fraud model stops flagging a pattern; editing documents in a data lake | Write access control, versioned datasets with hashes, diff review |
| Alter labels or annotations | Flipped labels from a compromised annotation account or vendor | Annotator identity, agreement checks, sampled re-labelling |
| Manipulate feedback and data-collection processes | Coordinated user feedback in online or reinforcement learning loops | Rate limits per source, trust weighting, holding feedback before it enters training |
Related ATLAS techniques sit around it: AML.T0010 AI Supply Chain Compromise for poisoned external datasets, AML.T0018 Manipulate AI Model for direct tampering with weights, AML.T0059 Erode Dataset Integrity for degrading data without a training goal, and AML.T0070 RAG Poisoning when the target is retrieval content rather than training data. RAG poisoning has its own guide, the MITRE ATLAS AML.T0071 False RAG Entry Injection.
Cases in ATLAS
- AML.CS0002 VirusTotal Poisoning: McAfee Advanced Threat Research noticed an unusual rise in reports of one ransomware family. Many samples had been submitted to a virus-sharing platform in a short time, all with the same compile time and 74 to 98 percent code similarity, produced with a metamorphic code tool. Several vendors began classifying them as the family even though most would not run, poisoning data that security models learn from. Lesson: crowd-sourced training feeds need submission-source and burst checks.
- AML.CS0009 Tay Poisoning: Microsoft wrote in March 2016 that within the first 24 hours of the Tay chatbot's launch a coordinated group exploited a vulnerability, leading Tay to post highly inappropriate content, and Tay was taken offline. ATLAS records abuse of a "repeat after me" function that skewed what Tay learned from interactions. Lesson: feedback loops that learn from users need gating, rate limits and the ability to roll back.
- AML.CS0025 Web-Scale Data Poisoning, Split-View Attack: the Carlini et al. paper showed that datasets distributed as lists of URLs can be poisoned by buying expired domains, because the content annotators saw differs from what later users download. It reports that 0.71% of LAION-400M images came from expired domains. The authors propose distributing cryptographic hashes of all content so clients can verify integrity, and randomising or delaying snapshot timing against frontrunning attacks on sources such as Wikipedia.
OWASP lists both split-view and frontrunning poisoning as examples under OWASP LLM Top 10 LLM04 Data and Model Poisoning Guide.
Where poisoning enters, and what to log
| Entry point | Detection signal | Log source |
|---|---|---|
| External datasets and URL lists | Content hash differs from the published hash; domain ownership changed since indexing | Ingestion job logs, hash verification results |
| Internal data lakes and feature stores | Writes to training paths by identities outside the data pipeline; edits to frozen dataset versions | Object storage data-access logs, data versioning commit history |
| Annotation and labelling | One annotator or vendor with unusual disagreement rate; label distribution shift in a batch | Labelling platform audit logs, quality metrics |
| User feedback and online learning | Bursts of similar feedback from few accounts; feedback content shift | Application logs, feedback queue metrics |
| Fine-tuning jobs | Training run using an unregistered dataset version; training loss pattern unlike previous runs | Experiment tracker, training job metadata |
| Evaluation | Clean accuracy unchanged but targeted test slices or trigger probes fail | Evaluation reports, model registry gates |
OWASP LLM04 recommends tracking data origin and transformations with tools such as CycloneDX ML-BOM, using data version control to detect manipulation, and monitoring training loss and model behaviour for signs of poisoning.
Mitigations from ATLAS, made practical
- AML.M0025 Maintain AI Dataset Provenance: ATLAS asks for a history of each dataset's source and every modification. Pin every training run to a dataset version with content hashes, and refuse runs that reference mutable paths.
- AML.M0007 Sanitize Training Data: ATLAS says to sanitise data before training and recurrently for active learning, filter ingested data and apply a content policy. Practical steps: source allow-lists, deduplication, near-duplicate and burst detection, and outlier checks per source.
- AML.M0008 Validate AI Model: test for backdoor triggers and monitor for concept and training data drift. Compare each new model with the previous approved one on fixed test slices, not only on overall accuracy.
- AML.M0023 AI Bill of Materials: record datasets and their versions in the AI BOM so a dataset reported as poisoned can be traced to every model trained on it. See the AI Supply Chain and AI-BOM Guide.
- AML.M0005 Control Access to AI Models and Data at Rest: limit who can write to training paths and labelling queues; separate data producers from pipeline identities.
- AML.M0001 Limit Model Artifact Release: ATLAS notes that published datasets can be targets; limit public detail about which datasets and sources feed production models.
Where identity fits
Direct poisoning needs write access, and most write access to training data belongs to non-human identities: ingestion jobs, ETL service accounts, labelling platform integrations and CI tokens that trigger fine-tuning. A stolen pipeline credential is the shortest route to AML.T0020. Give each pipeline stage its own identity with write rights only to its own output path, use short-lived credentials, and log every write to a training path. See the Model and dataset registry security guide for producer and consumer roles.
How to implement
- Inventory training data flows: for each production model, list datasets, sources, annotators, feedback loops and the identities that write to each.
- Freeze and hash: version every dataset used for training with content hashes; record the version in the experiment tracker and AI BOM.
- Restrict writes: remove broad write access to training paths; pilot on one pipeline, watch for broken jobs, keep the previous policy for rollback.
- Add ingestion checks: hash verification for external data, source allow-lists, burst and duplicate detection. Start in report-only mode, then block.
- Add a validation gate: new models must pass targeted slice tests and trigger probes against the last approved model before promotion. See the Fine-Tuning and LoRA Security Guide for fine-tuning specifics.
- Keep rollback ready: retain the previous approved model and dataset version so a suspected poisoning can be reverted quickly. The AI Model Poisoning Incident Playbook covers response.
How to test for poisoning and backdoors
- Planted canaries (AML.M0035): insert controlled poisoned records or a benign trigger into a representative, non-production pipeline and check which control catches it: ingestion checks, provenance, review, drift detection or the validation gate.
- Differential evaluation: compare new and previous models on fixed slices tied to business risk (high-value fraud patterns, safety refusals, named entities). A targeted change shows in a slice before it shows overall.
- Trigger probing: run candidate trigger sets (rare tokens, phrases, image patches relevant to your domain) and look for outputs that change sharply. Absence of a found trigger is not proof of absence.
- Data audits: sample and re-label a share of each annotation batch with a separate team; track disagreement per annotator.
The AI Red Teaming Guide covers scoping and rules of engagement.
Measuring it
- Share of production models whose training datasets are versioned with hashes and recorded in the AI BOM (target 100%).
- Number of identities with write access to training data paths, tracked downward.
- Share of planted canaries caught before promotion in red team exercises.
- Time to identify every model affected by a dataset reported as poisoned.
- Time to roll back to the previous approved model.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| MITRE ATLAS | AML.T0020 (tactic AML.TA0006 Persistence); AML.M0001, M0005, M0007, M0008, M0023, M0025, M0035 | The technique and its linked mitigations |
| OWASP Top 10 for LLM Applications 2025 | LLM04:2025 Data and Model Poisoning | Same risk for LLMs, including pre-training, fine-tuning and embeddings |
| NIST AI 100-2 E2025 | Adversarial Machine Learning taxonomy (poisoning attacks) | Terminology for poisoning attacks and mitigations |
| NIST SP 800-53 Rev 5 | SR-4 Provenance; SI-7 Software, Firmware, and Information Integrity; SI-10 Information Input Validation; AC-6 Least Privilege; SA-11 Developer Testing and Evaluation | Provenance, integrity, input checks, write scoping, validation |
| NIST CSF 2.0 | ID.AM-07; PR.DS-01; DE.CM-09 | Data inventories, data integrity at rest, monitoring |
Common failure patterns
- Training from mutable locations: jobs read "latest" folders or live URLs, so nobody can say what data a model saw.
- Deleting the data and keeping the model: poisoned records are removed but the model trained on them stays in production, which is exactly the persistence ATLAS describes.
- Accuracy-only validation: a backdoored model can keep overall accuracy while failing on trigger inputs or narrow slices.
- Unchecked feedback loops: user feedback flows straight into training, the Tay pattern.
- Shared pipeline credentials: one service account writes every dataset, so a single leak opens every training path.
Practitioner checklist
- Version and hash every training and fine-tuning dataset; record versions in the AI BOM.
- Verify hashes of external datasets at download.
- Scope write access to training paths per pipeline identity.
- Gate user feedback and online learning with rate limits and review.
- Require slice tests and trigger probes before model promotion.
- Keep the previous approved model and data version for rollback.
- Run planted-canary exercises at least yearly.
- Be able to list every model trained on a given dataset version.
Related ATLAS guides: AML.T0070 RAG Poisoning and AML.T0115 Publish Poisoned AI Artifacts.
Standards and references
- MITRE ATLAS: AML.T0020 Training Data Poisoning
- MITRE ATLAS: AML.M0025 Maintain AI Dataset Provenance
- OWASP: LLM04:2025 Data and Model Poisoning
- NIST: AI 100-2 E2025 Adversarial Machine Learning, A Taxonomy and Terminology of Attacks and Mitigations
- Carlini et al.: Poisoning Web-Scale Training Datasets is Practical
- Microsoft: Learning from Tay's introduction
- NIST: SP 800-53 Rev. 5 Security and Privacy Controls
Related NHI Mgmt Group resources: AI Data Poisoning and Backdoors Guide · OWASP LLM Top 10 LLM04 Data and Model Poisoning Guide · AI Model Poisoning Incident Playbook · MITRE ATLAS Mitigations Guide · MITRE ATLAS Guide