OWASP LLM04 Data and Model Poisoning Guide

If an attacker can change the data a model learns from, or swap the model file you load, every control downstream of the model is working on a corrupted foundation. OWASP lists this as LLM04:2025 Data and Model Poisoning in the Top 10 for LLM Applications, and classes it as an integrity attack on pre-training, fine-tuning and embedding data, plus the model artefacts themselves. This guide reads the OWASP entry clause by clause, turns each mitigation into something an engineering team can build and evidence, and shows how the risk maps to MITRE ATLAS, NIST and ISO controls. It is written for AppSec, ML platform and AI governance teams who fine-tune models, run retrieval pipelines or pull open-weight models from public hubs. For the attack mechanics in depth (clean-label attacks, sleeper agents, serialisation exploits), see the AI Data Poisoning and Backdoors Guide.

Key takeaways

Poisoning is a write-access problem first. Almost every route OWASP describes depends on someone being able to add or alter training, fine-tuning or retrieval data, so the identities holding that write access are part of the attack surface.
Model files are code until proven otherwise. OWASP explicitly calls out malicious pickling in shared repositories, where loading a model can execute an attacker's code before any evaluation runs.
Backdoors are built to pass your tests. A triggered backdoor leaves normal behaviour untouched, so a clean evaluation score is evidence of quality, not evidence of integrity.
Small amounts of data are enough. Research published in October 2025 by Anthropic with the UK AI Security Institute and The Alan Turing Institute found that around 250 malicious documents planted a backdoor in models from 600M to 13B parameters.
Provenance, versioning and rollback are the controls that age well. Detection improves slowly; knowing exactly which data and which model version produced a behaviour, and being able to revert, works against attacks you have not seen yet.

What it is and who it applies to

OWASP defines data poisoning as the manipulation of "pre-training, fine-tuning, or embedding data" to introduce vulnerabilities, backdoors or biases. The 2025 entry names four lifecycle stages where this can happen: pre-training on general data, fine-tuning for specific tasks, embedding (turning text into vectors for retrieval) and transfer learning. It also widens the scope beyond data to the model artefact: models distributed through shared repositories or open-source platforms can carry malware, for example through malicious pickling, and poisoning can implant a backdoor that stays dormant until a trigger appears.

The official risk name, confirmed on genai.owasp.org, is LLM04:2025 Data and Model Poisoning. In the 2023/24 edition the equivalent entry was Training Data Poisoning; the rename reflects the added model-artefact scope. Where it sits relative to the other nine risks is covered in the OWASP Top 10 for LLM Applications Guide.

Your position	Does LLM04 apply?	Where your exposure sits
Consumer of a hosted model API, no customisation	Partly	The provider owns pre-training integrity; you own any retrieval corpus, prompt templates and the data you send for provider-side fine-tuning
RAG application owner	Yes	Embedding and retrieval data: who can write to the knowledge base, connectors and vector store (overlaps LLM08)
Fine-tuner (hosted or self-run)	Yes, fully	Fine-tuning datasets, labelling vendors, feedback loops that turn user conversations into training data
Open-weight model downloader	Yes, fully	Model files from public hubs, adapters, tokenisers and conversion scripts (overlaps LLM03)
Pre-trainer	Yes, fully	Web-scale corpora, crawled sources, expired domains and dataset mirrors

LLM04 sits next to two neighbouring entries. Compromised third-party models and packages are the core of LLM03 Supply Chain, and poisoned retrieval content overlaps LLM08 Vector and Embedding Weaknesses. Rule of thumb: if the question is "where did this artefact come from", start with the OWASP LLM Top 10 LLM03 Supply Chain Guide; if it is "what did the model learn or retrieve, and who could change it", it is LLM04.

Structure and key requirements

The OWASP entry has four parts: a description, five common examples, ten prevention and mitigation strategies, and five example attack scenarios. The examples are worth reading precisely, because two are not malicious at all.

The five common examples

Training-time manipulation: attackers introduce harmful data during training to bias outputs, with OWASP naming split-view data poisoning (content behind an indexed dataset URL is changed later) and frontrunning poisoning (inserting content just before a periodic snapshot is taken).
Direct injection into the training process: an attacker with access to the pipeline adds content that degrades output quality.
Users supplying sensitive data: users unknowingly inject sensitive or proprietary information during interactions, which can surface in later outputs once conversations feed training.
Unverified training data: data accepted without validation raises the chance of biased or erroneous output.
Missing resource access restrictions: pipelines that can reach any source will ingest unsafe data.

The point about unverified web data is not theoretical. The 12,000 live secrets found in Common Crawl showed that a widely used training corpus carried working API keys and passwords: anything published on the open web can end up in a corpus, whether it is a credential or deliberately planted content.

The ten mitigations, translated into engineering requirements

#	OWASP mitigation (paraphrased)	What to build	Evidence it exists
1	Track data origins and transformations (OWASP CycloneDX, ML-BOM); verify legitimacy at every stage	An ML-BOM per model release listing base model digest, datasets with versions and hashes, adapters and transformation scripts	ML-BOM stored with each release and diffable between versions
2	Vet data vendors; validate outputs against trusted sources	Supplier review for labelling and data vendors; a trusted reference set for output comparison	Vendor assessments; reference-set results per release
3	Sandbox exposure to unverified sources; anomaly detection to filter adversarial data	Quarantine stage for new data with deduplication, outlier and near-duplicate checks before promotion	Quarantine logs with accept and reject counts
4	Use specific datasets for fine-tuning per use case	Separate, scoped fine-tuning sets instead of one shared pool	Dataset register mapping datasets to models and use cases
5	Infrastructure controls so the model cannot reach unintended sources	Egress allowlists for training jobs and crawlers; read-only mounts for approved data	Network policy and job configuration in version control
6	Data version control (OWASP names DVC)	Content-addressed dataset versions; training jobs reference versions, never "latest"	Every training run records dataset version and hash
7	Keep user-supplied information in a vector database so it can be adjusted without retraining	Retrieval for volatile or user-provided knowledge, with delete and re-index paths	Tested removal procedure and timing
8	Red team campaigns and adversarial testing	Trigger-search and behaviour-diff testing before release	Red team reports per major release
9	Monitor training loss and model behaviour; thresholds for anomalous output	Training telemetry alerts; production output drift monitoring	Alert definitions and triage records
10	RAG and grounding at inference to reduce hallucination	Grounded answers with citations to approved sources	Groundedness metrics (see LLM09)

Mitigation 7 has a trade-off: knowledge in retrieval is easier to correct, but the poisoning surface moves into the vector store, which then needs the write controls below. See the RAG Security Guide.

The five attack scenarios

OWASP's scenarios cover biased outputs, unfiltered toxic data, falsified documents planted by a competitor, misleading data inserted through prompt injection, and a backdoor trigger enabling authentication bypass, data exfiltration or hidden command execution. The last is the most underrated: in an agentic system, a backdoor that changes which tool gets called is an access control failure, not a content quality issue.

Identity and NHI requirements

Poisoning needs write access to something the model learns from or loads. In most organisations those write paths are held by non-human identities: pipeline service accounts, storage access keys, connector tokens, hub publishing tokens and labelling vendor accounts. Treat them as part of the LLM04 control set.

Write path	Typical identity	Requirement
Training and fine-tuning data buckets	Cloud service account or storage key used by ingestion jobs	Separate writer and reader identities; training jobs read-only; no long-lived keys where workload identity is available
Vector store and knowledge base	Connector OAuth tokens, indexer service accounts	Write scoped to named collections; human edits to source systems logged with author
Model registry and hub organisations	Publishing tokens, CI/CD credentials	Fine-grained, write-scoped tokens per repository; signing in CI, not on laptops
Labelling and data vendors	External user accounts or API keys	Time-bound access, per-vendor identities, upload to quarantine only
Feedback loops (thumbs up and down, conversation logs)	Application service account writing to training stores	Feedback is a candidate dataset, never written straight to an approved one

Prefer federated workload identity: training and ingestion jobs that use short-lived federated credentials leave no static key to steal from a notebook or repository. See the Workload Identity Federation Guide.
Scope pipeline credentials: the CI/CD identity that signs and publishes a model should not also be able to modify the datasets it was trained on. See the CI/CD Pipeline Identity Security Guide.
Inventory the service accounts: every notebook or job identity with write access needs an owner. See the Service Account Security Guide.

Implementation steps

Inventory models, datasets and write paths. List every model in production, its base model and source, every dataset used for fine-tuning and every retrieval corpus, and the identities that can write to each.
Gate model intake. Allow only non-executable weight formats (such as safetensors) from external sources, or load legacy formats only in an isolated sandbox with no credentials and no network. Pin downloads to a commit or digest, not a repository name. Hugging Face's own documentation warns that its pickle import scan is "not 100% foolproof", so treat scanning as one layer, not a gate on its own.
Version and hash datasets. Move fine-tuning and evaluation data into content-addressed versions, and make every training job record the dataset version, the base model digest and the code commit.
Add a quarantine stage. New data, including vendor deliveries and user feedback, lands in quarantine. Run deduplication, schema checks, outlier detection and secret scanning there, then promote by an approved job, not by hand.
Lock down write access. Apply the identity requirements above: separate writer and reader identities, read-only training jobs, scoped connector tokens, and alerts on writes outside the pipeline.
Build a release evaluation. Maintain a trusted held-out set plus targeted behaviour tests (refusals, tool-selection, security-sensitive answers). Compare each candidate model with the current production model and block release on unexplained regressions or behaviour changes.
Red team for triggers. Before major releases, test for trigger-dependent behaviour across unusual tokens, languages and formats. NIST AI 600-1 suggested action MS-2.7-007 calls for red teaming against ML attacks including data poisoning.
Sign and record releases. Sign model artefacts and their ML-BOM in CI and verify the signature at deployment. A signature proves origin and integrity in transit; it does not prove the data was clean, which is why steps 3 to 7 still matter.
Pilot, then enforce. Run intake gates and evaluation gates in report-only mode for one release cycle, review what they would have blocked, tune thresholds, then switch to enforcing.
Prove rollback. Keep the previous signed model and dataset versions deployable and rehearse a rollback, including re-indexing a retrieval corpus to a known-good snapshot.

Decision guidance

Retrieval or fine-tuning for volatile knowledge: OWASP's mitigation 7 favours retrieval, because a poisoned document can be deleted and re-indexed in minutes while a poisoned fine-tune needs a retrain or rollback. Choose fine-tuning for style and task behaviour, retrieval for facts that change.
Small organisations: focus on steps 1, 2, 3, 5 and 10. Safe formats, pinned versions, scoped write access and a working rollback cover most of the realistic risk.
Large organisations and pre-trainers: add the quarantine pipeline, ML-BOMs for every release, formal vendor assessment and a standing red team. Corpus-level defences against split-view and frontrunning poisoning (hash checks on crawled content, unpredictable snapshot timing) belong here.

Evidence and assessment

Assessors and internal audit will ask whether the mitigations exist and whether they work. The table lists what to log and what to keep.

Area	What to log or monitor	Signal worth an alert
Data stores	Object writes, deletes and permission changes on training, evaluation and retrieval stores	Writes by an identity that is not the ingestion pipeline; bulk changes outside a change window
Training runs	Dataset versions, base model digest, loss curves, evaluation results	Loss or evaluation deviation beyond threshold against the previous run with the same configuration
Model intake	Source URL, revision, digest, format, scan results	Executable format from an external source; unpinned revision; failed signature verification
Production behaviour	Output samples, refusal rates, tool-call distribution	Sudden shift in tool choice or refusal behaviour after a model or index update

Metrics

Provenance coverage: percentage of production models with a complete ML-BOM (base model digest, dataset versions, adapters).
Safe-format rate: percentage of externally sourced model artefacts loaded in a non-executable format or inside an isolated sandbox.
Unexpected writes: number of writes to approved datasets or vector stores by identities other than the promotion pipeline, per month (target zero).
Evaluation gate coverage: percentage of model releases that ran the full release evaluation and trigger tests before deployment.
Rollback time: measured time to restore the previous model or a known-good index in the last rehearsal.

Mapping to other frameworks

All IDs below were checked against the current MITRE ATLAS 5.6.0, NIST SP 800-53 Rev. 5 and NIST CSF 2.0 datasets. OWASP itself points to ATLAS AML.T0018, the NIST AI RMF and OWASP ML07:2023 Transfer Learning Attack.

Framework	Reference	How it relates to LLM04
MITRE ATLAS	AML.T0020 Poison Training Data; AML.T0019 Publish Poisoned Datasets	Training and fine-tuning data poisoning, including planting content in public datasets
MITRE ATLAS	AML.T0018 Manipulate AI Model (AML.T0018.000 Poison AI Model); AML.T0058 Publish Poisoned Models	Backdoored or altered model artefacts
MITRE ATLAS	AML.T0010.002 Data and AML.T0010.003 Model (AI Supply Chain Compromise); AML.T0011.000 Unsafe AI Artifacts	Compromised data and model sources; executable model files
MITRE ATLAS	AML.T0070 RAG Poisoning; AML.T0059 Erode Dataset Integrity	Embedding and retrieval stage poisoning
NIST SP 800-53 Rev. 5	SI-7, SR-3, SR-4, SR-11, CM-14, SR-6	Integrity of software and information, supply chain controls, provenance, component authenticity, signed components, supplier assessment
NIST SP 800-53 Rev. 5	SA-11, SI-4, RA-3	Developer testing and evaluation, monitoring, risk assessment
NIST CSF 2.0	GV.SC-01, GV.SC-05, ID.AM-07, PR.DS-01, DE.CM-09	Supply chain programme and contract requirements, data inventories, data-at-rest integrity, monitoring of software and data
NIST AI 600-1	MS-2.5-005, MS-2.7-007, MP-2.1-001, MS-2.7-001	Verify training and TEVV data provenance; red team for data poisoning; data origin and lineage; assess backdoors and compromised dependencies
NIST AI 100-2 E2025	Adversarial Machine Learning taxonomy	Shared terminology for poisoning attack classes and mitigations
EU AI Act	Article 15(5)	High-risk AI systems need measures against attacks that manipulate the training data set (data poisoning) or pre-trained components (model poisoning)
ISO/IEC 27001:2022 Annex A	A.5.21, A.8.25, A.8.16	ICT supply chain security, secure development life cycle, monitoring activities
ISO/IEC 42001:2023	AI management system	Management-system home for data quality and provenance processes

On the EU AI Act, the obligations in Article 15 apply to high-risk systems from a later date than the Act's earlier provisions; artificialintelligenceact.eu currently lists 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems. Confirm against the official text; see the EU AI Act High-Risk AI Systems Guide.

Common pitfalls

Loading models like data. OWASP names malicious pickling for a reason: until the format is non-executable or the load is sandboxed, the file is code.
Trusting a name instead of a digest. Pulling "org/model" at its latest revision means any change upstream, including a compromised publishing token, reaches you automatically. Pin revisions and verify signatures. See the Model Supply Chain Guide.
Feedback loops with no quarantine. Conversation logs and user ratings piped straight into fine-tuning data give every user a write path into the model, and OWASP's third example (users injecting sensitive data) arrives with it.
Open write access to the knowledge base. A wiki or shared drive that everyone can edit, indexed into a RAG corpus, lets any account plant instructions or false facts. Scope connectors and log edits with authors.
Treating a passing benchmark as proof of integrity. Backdoors are designed to leave benchmark scores unchanged. Behaviour-diff and trigger testing answer a different question from accuracy testing.
No rollback path. If the previous model, dataset version and index snapshot were not kept, a confirmed poisoning event becomes a full retrain under pressure.

Practitioner checklist

Inventory every production model, base model source, fine-tuning dataset and retrieval corpus, with an owner for each.
List every identity with write access to training, evaluation, retrieval and model registry stores, and remove what is not needed.
Allow only non-executable weight formats from external sources, or sandbox the load with no credentials or network.
Pin model and dataset downloads to digests or commits; verify signatures at deployment.
Version and hash datasets; record dataset version, base model digest and code commit for every training run.
Route vendor data and user feedback through a quarantine stage with deduplication, outlier and secret scanning.
Run a release evaluation and trigger tests against the current production model before each deployment.
Produce an ML-BOM (for example in CycloneDX) for each model release and store it with the artefact.
Alert on writes to approved data stores by identities other than the promotion pipeline.
Rehearse rollback of a model and a retrieval index, and record the time it took.

OWASP LLM Top 10 LLM04 Data and Model Poisoning Guide: Protecting Training and Fine-Tuning Data