LLM data poisoning is the deliberate insertion of malicious examples into training or fine-tuning data so the model learns unintended behaviour. The attack can create hidden backdoors, bias outputs, or change responses when a specific trigger appears during inference.
Expanded Definition
LLM data poisoning is a supply chain attack on model training or fine-tuning data. The attacker injects crafted samples so the model internalises malicious patterns, backdoors, or skewed associations that later influence inference. In practice, this can affect pretraining corpora, supervised instruction data, preference data, retrieval indexes used for adaptation, or internal datasets assembled for domain tuning. The security issue is not only corrupted data quality. It is the loss of trust in what the model has learned and why it behaves a certain way.
Definitions vary across vendors on whether poisoning must be targeted, persistent, or trigger-based, but the common security meaning is deliberate manipulation of model-shaping data. This aligns closely with guidance in the NIST AI Risk Management Framework and the NIST AI 600-1 Generative AI Profile, both of which emphasise data governance, provenance, and robustness across the AI lifecycle. LLM poisoning is distinct from prompt injection because it alters what the model learns, not just what it sees at runtime. The most common misapplication is treating poisoned outputs as ordinary model error, which occurs when teams fail to trace the behaviour back to compromised training or fine-tuning data.
Examples and Use Cases
Implementing defences against LLM data poisoning rigorously often introduces tighter dataset controls, slower iteration, and more review overhead, requiring organisations to weigh model development speed against provenance assurance.
- A malicious contributor slips trigger-based examples into a public dataset, causing a fine-tuned assistant to reveal unsafe content when a specific phrase appears.
- A contractor inserts biased labels into preference-tuning data, shifting the model toward one class of answers and degrading decision quality in production.
- An attacker compromises a data pipeline and alters instruction examples before training, creating a hidden backdoor that activates only for a rare token pattern.
- A team ingests scraped web content without provenance checks, and the model absorbs poisoned samples that distort domain-specific answers over time.
- Security teams map the attack to adversarial AI threats described in the MITRE ATLAS adversarial AI threat matrix and use the OWASP Agentic AI Top 10 to prioritise data integrity controls.
These use cases are especially relevant where fine-tuning data, retrieval sources, or human feedback pipelines are assembled from many contributors and weakly governed inputs.
Why It Matters for Security Teams
LLM data poisoning matters because it creates a model integrity problem that can survive normal testing. A poisoned model may appear safe during evaluation, then fail only when a trigger, topic, or user pattern appears in production. That makes detection difficult and incident response expensive. For security teams, the core task is to protect dataset provenance, validate ingestion paths, and establish review controls for training and fine-tuning sources. The OWASP Top 10 for Agentic Applications 2026 and CSA MAESTRO agentic AI threat modeling framework both reinforce that AI systems need lifecycle threat modeling, not just runtime filtering.
This term also intersects with identity governance when non-human identities, service accounts, or agent pipelines can write to training stores or feedback queues. If those identities are overprivileged, compromised, or poorly attested, data poisoning becomes easier to stage and harder to unwind. Organisations typically encounter the full impact only after a model starts behaving inconsistently or a malicious trigger is discovered in production, at which point data poisoning becomes operationally unavoidable to address.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, MITRE ATLAS and CSA MAESTRO address the attack and risk surface, while NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI RMF defines governance, map, measure, and manage practices for AI data risk. | |
| NIST AI 600-1 | The GenAI profile covers lifecycle risks including data quality and robustness concerns. | |
| OWASP Agentic AI Top 10 | OWASP Agentic AI highlights supply chain and data integrity threats affecting model behaviour. | |
| MITRE ATLAS | ATLAS catalogs adversarial AI techniques including poisoning and backdoor-related attacks. | |
| CSA MAESTRO | MAESTRO frames agentic AI threats across the model, data, and orchestration layers. |
Apply AI RMF governance to track training data provenance, review risk, and document model lineage.
Related resources from NHI Mgmt Group
- Why do LLM applications create new data leakage risks for identity teams?
- How should security teams govern LLM applications that call tools and data sources?
- How should security teams govern sensitive data in LLM workflows?
- Who is accountable when an LLM leaks data after following malicious instructions?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on August 18, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
