Join our Newsletter — 33% off our NHI Course

AI Poisoning Attacks

AI poisoning attacks are attempts to corrupt an AI system by feeding it bad training data, prompts, or retrieved content so it learns or acts incorrectly. In practice, attackers target model training, fine-tuning, or RAG inputs to insert false patterns, bias outputs, weaken controls, or trigger unsafe behavior.

What AI poisoning attacks target

ai poisoning attacks aim at the data path that teaches or steers a model. The attacker is not necessarily trying to break the model directly, but to corrupt what it learns from training sets, fine-tuning corpora, prompts, or retrieved context.

That makes the attack broad in practice: a poisoned sample can shift model behaviour subtly, while a larger poisoning campaign can bias outputs, weaken safety filters, or insert a trigger that produces unsafe behaviour later. The same pattern applies whether the target is model training, instruction tuning, or retrieval-augmented generation.

Why poisoning is dangerous in AI systems

Poisoning is dangerous because the corruption is often absorbed as if it were legitimate data. Once bad data is inside the learning or retrieval pipeline, the system may repeat the attacker’s pattern with high confidence, which makes the resulting failure look like a normal model decision rather than an intrusion.

That is especially problematic in systems that continuously ingest new content or adapt over time. A poisoned dataset, prompt cache, or knowledge source can create persistent errors, cause targeted misclassification, or push the system toward unsafe recommendations without leaving an obvious signature in the final output.

For broader context on adversarial AI patterns, MITRE ATLAS adversarial AI threat matrix maps poisoning-style techniques to the kinds of tactics defenders can hunt for.

How poisoning enters the training and retrieval lifecycle

Poisoning can occur at several points. Training data poisoning alters the examples used to build the model. Fine-tuning poisoning changes how a model behaves in a narrower domain. RAG poisoning corrupts the retrieved content the model relies on at answer time. In each case, the attacker is exploiting trust in the data supply chain.

Defenders should think about provenance, ingestion controls, and review boundaries rather than assuming “internal” data is safe. Poisoning often succeeds when content arrives from many sources, when labeling is weak, or when retrieved documents are treated as authoritative without validation.

NHIMG’s AI Data Poisoning and Backdoors Guide covers training, fine-tuning and RAG poisoning, plus provenance and detection considerations.

What organisations should validate and monitor

Exposure assessment starts with knowing which data sources influence the model, who can modify them, and where untrusted content can enter the pipeline. The key question is not only whether the model is accurate today, but whether its upstream inputs can be manipulated tomorrow.

Detection is usually indirect. Teams look for anomalous samples, unusual label patterns, sudden changes in model behaviour, unexpected retrieved content, or outputs that reflect a hidden trigger. After a suspected poisoning event, the safest response is often to treat the affected model, dataset, or retrieval corpus as compromised until provenance can be re-established.

If your environment includes agent memory or persistent context stores, AI Agent Memory Poisoning Guide is useful for understanding how corrupted context can persist across sessions.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

MITRE ATLAS addresses the attack surface, NIST AI RMF, NIST SP 800-53 Rev 5 and OWASP ASVS set the technical controls, and ISO/IEC 42001:2023 defines the regulatory obligations.

Framework Control / Reference Relevance
MITRE ATLAS Adversarial Machine Learning Techniques Poisoning is a core adversarial AI technique covered by ATLAS.
Recommendation — Map poisoning patterns to ATLAS techniques and hunt for compromised data sources.
NIST AI RMF Govern, Map, Measure, Manage AI poisoning is an AI risk-management issue across the model lifecycle.
Recommendation — Apply AI RMF to govern data provenance, measure model drift, and manage poisoning risk.
NIST SP 800-53 Rev 5 SI-7 — Software, Firmware, and Information Integrity Poisoned training or retrieval inputs threaten information integrity in AI pipelines.
Recommendation — Use SI-7 to validate AI inputs and detect integrity degradation in model data flows.
OWASP ASVS V15 — Secure Coding and Architecture Poisoning reflects architecture weaknesses in how AI inputs are trusted and handled.
Recommendation — Design AI data pipelines to validate inputs, isolate sources, and prevent trust abuse.
ISO/IEC 42001:2023 4 — Context of the organization AI poisoning is managed through organisation-wide AI governance and risk context.
Recommendation — Establish governance for dataset provenance, source approval, and model-change accountability.

Practitioner Guidance

Why practitioners should care: Poisoning is a control-plane problem as much as a model-quality problem. If you cannot trust the data supply chain, you cannot trust the behaviour of the system that learned from it.

Governance implication: Assign ownership for dataset provenance, retrieval-source approval, and rollback decisions before the model goes live. In practice, that means treating training sets, tuning corpora, and retrieval indexes as governed assets, not disposable inputs.

For organisations building broader AI control coverage, AI Security Posture Management (AI-SPM) Guide provides a useful operating model for discovery, posture, and attack-path thinking across AI assets.

Practitioner takeaway: The most effective defence is to reduce trust in unvetted inputs, increase provenance confidence, and monitor for behaviour changes that suggest the model has already absorbed poisoned data.