AI Data Poisoning and Backdoors Guide

A model is only as trustworthy as the data and artefacts it was built from, and an attacker who can influence either can plant behaviour that ordinary testing will not find. Data poisoning corrupts training, fine-tuning or retrieval data; backdoors (also called trojans) make a model behave normally until a trigger appears; and model supply chain attacks deliver poisoned or malicious model files through hubs and registries. This guide explains how these attacks work at each stage of the AI lifecycle, why backdoors survive safety training, and which controls (provenance, curation, signing, scanning, evaluation and monitoring) reduce the risk. It is written for ML platform owners, AI security engineers and the security architects who approve AI data pipelines. It describes attacks only at the level needed to defend against them.

Key takeaways

• Poisoning is an integrity attack on the data supply chain. NIST, OWASP and MITRE ATLAS all treat it as such, and ATLAS marks Training Data Poisoning (AML.T0020) as "Realized", meaning it has been seen in real systems.
• The number of poisoned samples needed can be small. Research by Anthropic, the UK AI Security Institute and the Alan Turing Institute found that as few as 250 malicious documents produced a backdoor in models from 600M to 13B parameters.
• Backdoors can survive safety training. The Sleeper Agents research showed backdoored behaviour persisting through standard safety techniques, with adversarial training sometimes teaching the model to hide its trigger better.
• Model files can be malware. Pickle-based formats run code when loaded, so a downloaded model is executable content until proven otherwise. Prefer safetensors, scan files and verify signatures.
• Provenance beats detection. Detection research has real limits, so the strongest controls are knowing where every dataset and model came from, who could change it, and whether it is the version you approved.

How poisoning and backdoors work

NIST AI 100-2 E2025 defines poisoning attacks as "adversarial attacks during the training stage" and separates data poisoning, where "an adversary controls a subset of the training data", from model poisoning, where "the adversary controls the model and its parameters". The attacker's goal decides the type:

OWASP LLM04:2025 Data and Model Poisoning extends this to large language models: "Data poisoning occurs when pre-training, fine-tuning, or embedding data is manipulated" to add vulnerabilities, backdoors or biases. It notes that backdoors "may leave the model's behavior untouched until a certain trigger causes it to change", which is why they are hard to test for. The consequences OWASP lists for a triggered backdoor include authentication bypass, data exfiltration and hidden command execution when the model drives downstream actions.

Where poisoning enters the AI lifecycle

• Web-scale data: Carlini and colleagues showed split-view and frontrunning poisoning were practical, estimating they could have poisoned 0.01% of LAION-400M or COYO-700M for about USD 60, and that the attacks could affect 10 popular datasets. MITRE records this as case study AML.CS0025 "Web-Scale Data Poisoning: Split-View Attack".
• Online learning: systems that learn from users are exposed by design. OWASP cites MITRE ATLAS case study AML.CS0009 Tay Poisoning, where a chatbot learned from coordinated abusive input.
• RAG is not training, but behaves like it: a planted document changes answers without touching weights. Mechanics and controls for retrieval poisoning are covered in the Prompt Injection Guide and the AI Agent Memory Security Guide.

Sleeper agents: why backdoors survive testing

The Sleeper Agents paper (Hubinger et al., January 2024) built proof-of-concept language models that wrote secure code when the prompt said the year was 2023 and inserted vulnerabilities when it said 2024. The authors found that "such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques", that persistence was greatest in the largest models, and that adversarial training "can teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior". Their conclusion was that standard techniques could "create a false impression of safety".

The October 2025 study by Anthropic with the UK AI Security Institute and the Alan Turing Institute adds a second point: the number of poisoned documents needed did not grow with model or dataset size in their experiments. As few as 250 documents installed a backdoor that made models output gibberish on a trigger phrase. The authors stress that this was a narrow, low-harm backdoor and that "it remains unclear how far this trend will hold" for larger models.

What this means for defenders:

• A model passing your safety and accuracy evaluation is not evidence that it has no backdoor.
• Percentage-based thinking ("an attacker would need to control a large share of our data") is unsafe; a small absolute number of samples may be enough.
• Third-party models, including fine-tunes of reputable base models, need provenance controls as well as testing.

Model supply chain: serialisation and model hubs

Hugging Face's documentation states that pickle "allows more flexibility by giving the ability to users to run python code at de-serialization time", through opcodes such as GLOBAL, STACK_GLOBAL and REDUCE. Loading a pickled checkpoint from an untrusted source is therefore equivalent to running untrusted code with the loader's privileges, which on a training node often means cloud credentials and dataset access.

• Model signing in practice: the OpenSSF model signing project (v1.0 released April 2025, pip package model-signing) hashes every file in a model into a manifest and signs it as an in-toto statement in a DSSE envelope. Sigstore keyless signing is the default; public keys, certificates and PKCS #11 devices are also supported. The CLI is model_signing sign and model_signing verify, with the expected signer identity passed at verification. The Sigstore transparency log means "a rogue insider cannot release new models as if they are signed by the company".
• Hub-specific attacks: ATLAS release 2026.07 added case studies AML.CS0064 "Poisoned GGUF Templates: Inference-Time Supply Chain Attack" and AML.CS0065 "Model Namespace Reuse Supply Chain Attack", where a deleted or transferred publisher namespace is re-registered by someone else. Pin to a commit or digest, not a repository name.
• Signed commits on hubs help, but in Hugging Face's words this "does not guarantee that your file is safe, but it does guarantee the origin of the file". Use origin as an input to trust decisions, not as the decision.

For AI-BOMs, model registries and dependency controls, see the AI Supply Chain and AI-BOM Guide.

Defences: provenance, curation, evaluation and monitoring

The May 2025 joint guidance "AI Data Security: Best Practices for Securing Data Used to Train & Operate AI Systems" from NSA, CISA, FBI, ASD's ACSC, NCSC-NZ and NCSC-UK names three data risks: the data supply chain, maliciously modified data and data drift. Its headline recommendations are digital signatures to verify trusted data revisions, provenance tracking and trusted infrastructure.

Data provenance and curation

• Record origin and transformations for every dataset (source, licence, collection date, hash, every filter and transformation), using CycloneDX ML-BOM or an equivalent, as OWASP recommends. This is ATLAS mitigation AML.M0025 Maintain AI Dataset Provenance.
• Version datasets with content hashes (OWASP names DVC as one option) so any change between approval and training is detectable, and sign approved versions.
• Sanitise training data (AML.M0007): deduplicate, filter outliers and near-duplicates, check label consistency, and quarantine new sources until reviewed. NIST warns sanitisation "is not as effective against stealthy poisoning attacks", so treat it as one layer.
• Control write access to raw data buckets, labelling tools, feature stores and RAG sources. Most poisoning in enterprises needs write access to something you own.

Model evaluation and backdoor detection

• Validate models (AML.M0008) against held-out data you control, and compare behaviour with the base model on sensitive tasks.
• Backdoor detection techniques from NIST's taxonomy include trigger reconstruction (Neural Cleanse "used optimization to determine the most likely backdoor pattern"), model inspection followed by pruning or fine-tuning, and poison forensics to trace a backdoor to its training samples. NIST notes these "generally assume fixed backdoor patterns", which newer triggers evade.
• Red team for triggered behaviour (AML.M0035 AI Red Team): probe for outputs that change with dates, rare tokens, names or formatting, and for code generation that differs by context. See Red Teaming AI Agents for Identity Abuse.
• Robust training options where you train: ensembles, robust aggregation for federated learning, and differential privacy, accepting NIST's caution that clipping and differential privacy "usually decrease accuracy and do not provide complete mitigation".

Choosing controls by how you use AI

Smaller organisations usually sit in the first two rows: put effort into knowledge base write controls and model intake. Larger organisations that train should fund a data security function with the same rigour as source code management.

Implementation roadmap

1. Inventory (weeks 1 to 4): list every model, dataset, adapter and RAG source in use, where it came from and who can write to it. Record it in an AI-BOM.
2. Model intake gate: require a non-executable format or a sandboxed load, a scan and, where available, signature verification before a model reaches a shared registry. Run the gate in report-only mode for two to four weeks, review what it would have blocked, then enforce.
3. Dataset controls: move approved datasets to versioned, hashed, access-controlled storage; block training jobs from reading unregistered sources.
4. Sign your own artefacts: sign models and datasets your teams produce, and verify at deployment.
5. Evaluation baseline: build behavioural tests for each production model, including trigger probes, and run them on every new model or fine-tune.
6. Monitoring: add the signals below and route them to the SOC.
7. Rollback: keep the previous signed model and dataset versions deployable, so a suspected poisoning can be answered by reverting rather than retraining under pressure.

Where identity fits

Poisoning usually needs write access, and write access belongs to identities. The service accounts used by data pipelines, labelling vendors' accounts, model hub publishing tokens and CI credentials that push to registries are the practical attack path, more often than clever data crafting. Scope them to specific buckets and repositories, prefer short-lived workload identity over static tokens, and alert on writes from unexpected identities. The LiteLLM PyPI breach showed how a leaked publishing token lets an attacker ship a poisoned release of a widely trusted AI component. See the AI Infrastructure Workload Identity Guide.

Detection and measurement

• Data pipeline: writes to training, fine-tuning and RAG stores by identity; hash mismatches between approved and consumed datasets; sudden shifts in label or class distribution.
• Training: anomalous training loss curves (OWASP recommends thresholds on training loss and model behaviour); unexpected gradient norms in federated learning.
• Model intake: scan findings, signature failures, models loaded from outside the approved registry.
• Production: output drift against the baseline, clusters of unusual outputs on specific inputs, and user reports, logged with ATLAS mitigation AML.M0024 AI Telemetry Logging in mind.

Framework mapping

Common failure patterns

• Loading models like data: data scientists pull pickled checkpoints onto credential-rich notebooks. Treat model files as code.
• Trusting a name instead of a digest: pipelines that fetch "latest" from a hub inherit any upstream change, including namespace takeover.
• Open write access to knowledge bases: wikis, ticket systems and shared drives indexed for RAG let any user, or any compromised account, shape answers.
• Treating AI packages as low risk: the Mastra npm supply chain attack backdoored more than 140 AI framework packages through a forgotten contributor account.
• Relying on evaluation alone: a clean benchmark score says nothing about triggered behaviour, as the sleeper agent research shows.
• No last known good: without signed, retained prior versions, recovery from suspected poisoning means retraining from data you can no longer trust.

Practitioner checklist

• Maintain an AI-BOM covering every model, adapter, dataset and RAG source, with provenance.
• Require safe weight formats or sandboxed loading, and block trust_remote_code for unreviewed repositories.
• Scan model files and verify signatures at intake; pin models by digest or commit.
• Sign internally produced models and datasets and verify them at deployment.
• Store approved datasets versioned and hashed, and block training on unregistered data.
• Restrict and monitor write access to training data, labelling tools and RAG sources.
• Run behavioural and trigger-probing evaluations on every new model or fine-tune.
• Monitor training loss, output drift and data distribution shifts, with alerts to the SOC.
• Keep last known good models and datasets ready for rollback.

Standards and references

• NIST: AI 100-2 E2025 Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
• OWASP: LLM04:2025 Data and Model Poisoning
• NSA, CISA, FBI and partners: AI Data Security joint guidance
• MITRE: ATLAS Data
• OpenSSF and Sigstore: Model Transparency (model signing)
• Hugging Face: Pickle Scanning
• Hubinger et al.: Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
• Carlini et al.: Poisoning Web-Scale Training Datasets is Practical

Related NHI Mgmt Group resources: AI Supply Chain and AI-BOM Guide · LLM Security Guide · AI Agent Memory Security Guide · AI Governance Guide