AML.T0059 Erode Dataset Integrity: Keep Your Training Data Trustworthy and Usable

A dataset does not have to produce a backdoored model to be damaged: if enough of it is wrong, people stop trusting it and spend their time cleaning it. MITRE ATLAS tracks this as AML.T0059 Erode Dataset Integrity, in the Impact tactic (AML.TA0011), with maturity rated Demonstrated. ATLAS describes adversaries poisoning or manipulating portions of a dataset to reduce its usefulness, reduce trust in it and make users waste resources correcting errors. This guide is for data engineers, ML platform owners and AI security teams who curate, buy or download datasets: how the technique differs from training data poisoning, what the split-view research shows, which two mitigations ATLAS maps, and how to build integrity checks into the data pipeline. For the wider topic of poisoned data and backdoors, see the AI Data Poisoning and Backdoors Guide.

Key takeaways

• The dataset is the victim. AML.T0059 is about the lost value of the data asset itself: usefulness, trust and clean-up cost, whatever happens to any model trained on it.
• It usually arrives with poisoning. In the ATLAS split-view case study the same poisoned data both eroded the dataset (AML.T0059) and eroded model integrity (AML.T0031), so treat the two as one incident with two impacts.
• ATLAS maps two mitigations. AML.M0007 Sanitize Training Data and AML.M0025 Maintain AI Dataset Provenance.
• URL-list datasets are especially exposed. Research cited by ATLAS showed that expired domains in web-scale datasets could be bought and used to change what later users download.
• Hashes and versions are the cheapest defence. Recording a cryptographic hash of each item at collection time, and rejecting mismatches at download time, closes the split-view gap.

How the technique works

The ATLAS description is short: adversaries poison or manipulate portions of a dataset so that it becomes less useful, less trusted and more expensive to maintain. The manipulation can include changed samples, labels, annotations or metadata, the same targets that ATLAS lists for screening under AML.M0007. What distinguishes the technique is the goal. The adversary may not need any model to misbehave in a particular way: a dataset that teams no longer trust, that must be re-labelled, or that has to be withdrawn is already a loss.

Where datasets are exposed:

• Datasets distributed as URL lists: the content at each URL can change after the list was compiled.
• Snapshots of crowd-sourced content: material edited into a source just before a scheduled snapshot can be captured.
• Shared internal datasets and feature stores: anyone with write access can change records or labels.
• Labelling and annotation pipelines: outsourced or crowd-sourced labelling can introduce systematic errors.
• Feedback and active learning loops: user feedback fed back into datasets lets outsiders write to them indirectly.
• Retrieval corpora for LLM applications: knowledge bases used for retrieval are datasets too; see AML.T0070 RAG Poisoning for the retrieval-specific technique.

Telling it apart from related techniques

See AML.T0020 Training Data Poisoning, MITRE ATLAS AML.T0031 Erode AI Model Integrity and AML.T0115 Publish Poisoned AI Artifacts for those techniques.

Case study: split-view poisoning of web-scale datasets

ATLAS records one case study for this technique, AML.CS0025 Web-Scale Data Poisoning: Split-View Attack, based on the paper "Poisoning Web-Scale Training Datasets is Practical" by Carlini and colleagues. The ATLAS procedure runs as follows:

1. The researchers downloaded a web-scale dataset made of URLs pointing to individual data points (AML.T0002.000 Datasets).
2. They identified expired domains in the dataset and bought them (AML.T0008.002 Domains).
3. An adversary could then create poisoned data (AML.T0020) and host it on those domains (AML.T0115.000), with the researchers tracking requests to show the URLs were still being downloaded.
4. Future downloads of the dataset would contain the poisoned data points, eroding the dataset's integrity (AML.T0059).
5. Models trained on it would be poisoned; ATLAS notes as little as 0.01% of the data needed to be poisoned for a successful attack (AML.T0031).

The paper estimates that 0.01% of LAION-400M or COYO-700M could have been poisoned for about US$60, and also describes frontrunning poisoning, in which an attacker edits crowd-sourced content such as Wikipedia shortly before a snapshot is taken. The authors notified the maintainers of affected datasets.

The lesson for any organisation that builds on public data: the dataset you approved is not necessarily the dataset you download next month.

Controls

Maintain dataset provenance (AML.M0025)

• Record the source: for every dataset, store where it came from, who supplied it, the licence and the date acquired.
• Version every change: keep an immutable history of additions, deletions and label changes, with the identity that made each one. Data version control tools or object storage versioning both work.
• Hash at collection time: the split-view paper recommends that maintainers attach a cryptographic hash of the raw content obtained from each URL when the dataset is built, and that downloaders discard any item whose content does not match. The authors note this can reject benign changes such as re-encoded images, which is an acceptable cost for most uses.
• Delay and randomise snapshots of editable sources: against frontrunning, the paper proposes randomising snapshot order or freezing content, waiting for review to catch malicious edits, and only then releasing the snapshot.

Sanitise training data (AML.M0007)

• Screen on acquisition and update: ATLAS recommends screening when datasets are acquired or updated and before new data enters active learning workflows.
• Check samples, labels, annotations and metadata: look for label inconsistencies, anomalous samples, unexpected shifts in source or class distributions, suspicious duplication and backdoor triggers, compared with a trusted version.
• Quarantine, then remediate: hold suspicious records, remove malicious additions, correct labels or restore from a trusted version, then screen again before training.
• Trace to models: if affected data has already been used, identify the resulting models and decide on retraining or rollback, combined with AML.M0008 Validate AI Model.

ATLAS names the Adversarial Robustness Toolbox and Cleanlab among tools that support screening. See the Adversarial Robustness Toolbox Guide.

Restrict who can write

Datasets and feature stores are often writable by broad groups and by pipeline service accounts. Limit write access to named roles and specific pipeline identities, require review for label changes at scale, and log every write. Registry controls are covered in Model and dataset registry security.

Implementation steps

1. Inventory datasets: list every training, fine-tuning, evaluation and retrieval dataset, its owner, its source and who can write to it.
2. Baseline: take a versioned, hashed snapshot of each dataset and record its distribution statistics as the trusted reference.
3. Add checks to ingestion in report-only mode: run hash verification and screening on new or refreshed data, log findings without blocking, and tune thresholds for a few cycles.
4. Enforce: block training on data that fails hash or screening checks until reviewed; keep a documented override with an approver.
5. Plan rollback: make sure any dataset can be restored to a named trusted version and any model traced to the data version it used.
6. Review after each incident: feed confirmed manipulation patterns back into screening rules.

Decision guidance

Detection and measurement

• Hash mismatches: items whose content no longer matches the recorded hash, by source domain.
• Distribution drift: unexpected changes in class, source or label distributions between versions.
• Unusual writes: large label changes or bulk inserts by unexpected identities, or outside normal pipeline runs.
• Downstream complaints: rising data-quality tickets or rework time from data consumers, which is the direct measure of the harm ATLAS describes.

Metrics:

• Percentage of training datasets with recorded source, version history and owner.
• Percentage of externally sourced items with a verified content hash.
• Percentage of dataset writes made by approved pipeline identities.
• Time to restore a dataset to its last trusted version, tested at least yearly.
• Number of models that can be traced to the exact data version they were trained on.

Framework mapping

Common failure patterns

• Re-downloading by URL without hashes: the split-view gap stays open for every refresh.
• No trusted version to compare against: screening has no baseline, so manipulation looks like normal change.
• Broad write access to datasets: shared buckets and feature stores let any compromised account or pipeline alter data.
• Feedback loops without screening: user feedback becomes training data with no check on volume or source.
• No lineage from model to data: after a problem is found, nobody can tell which models need retraining.

Practitioner checklist

• Keep an inventory of datasets with owner, source, licence and write access.
• Store datasets in versioned storage with an immutable change history.
• Record content hashes for externally sourced items and reject mismatches.
• Screen samples, labels, annotations and metadata on every acquisition and update.
• Restrict writes to named roles and pipeline identities, and log every write.
• Record the data version used for every trained model.
• Test restoring a dataset to a trusted version.
• Track data-quality rework time as an impact measure.

Standards and references

• MITRE ATLAS: Erode Dataset Integrity
• MITRE ATLAS: Sanitize Training Data
• MITRE ATLAS: Maintain AI Dataset Provenance
• MITRE ATLAS: Web-Scale Data Poisoning: Split-View Attack
• Carlini et al.: Poisoning Web-Scale Training Datasets is Practical
• OWASP: LLM04:2025 Data and Model Poisoning
• NIST: AI 100-2 E2025 Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations

Related NHI Mgmt Group resources: AI Data Poisoning and Backdoors Guide · Data lineage, data quality and data contracts as security controls · Model and dataset registry security · MITRE ATLAS Guide