Poisoning Attack

A poisoning attack introduces malicious or misleading data into a model’s training process so the system learns the wrong patterns. This can reduce accuracy, introduce bias, or create hidden backdoors in behavior. The harm often appears later, after the model has already been deployed and trusted in production.

Expanded Definition

A poisoning attack is a deliberate effort to corrupt the data used to train or fine-tune a model so that the model internalises attacker-chosen patterns. In AI security, the term covers both obvious data tampering and subtle manipulation that is hard to distinguish from legitimate input at collection time. The goal may be to degrade performance, skew predictions toward a target outcome, or plant a backdoor that activates only under specific triggers.

Definitions are still evolving across vendors and research communities, but the core idea is consistent: the attacker is influencing what the model learns rather than attacking the model only at inference time. That makes poisoning attacks especially relevant to data pipelines, model supply chains, and any workflow that ingests third-party or user-generated content. MITRE’s MITRE ATLAS adversarial AI threat matrix is useful here because it catalogues adversary behaviours that target AI systems, even though the operational details vary by deployment.

The most common misapplication is treating all poor model outcomes as poisoning, which occurs when teams assume corruption even though the root cause is weak data quality, concept drift, or misconfigured labeling.

Examples and Use Cases

Implementing poisoning defences rigorously often introduces more review, provenance tracking, and filtering, requiring organisations to weigh model quality and speed against the cost of stricter ingestion controls.

• A spam classification model is retrained on attacker-inserted examples that make malicious messages appear benign.
• An image model is exposed to a small set of manipulated training samples that create a hidden trigger for misclassification.
• A retrieval or ranking system is tuned on contaminated feedback data, causing it to prioritise attacker-preferred content.
• A code assistant is fine-tuned on poisoned samples so it emits insecure patterns or follows a malicious backdoor prompt.
• A security team reviewing suspicious campaign activity correlates the model issue with broader adversary tradecraft, then checks MITRE ATT&CK Enterprise Matrix and current CISA cyber threat advisories for related intrusion patterns.

Poisoning also matters in agentic AI workflows where an assistant learns from logs, feedback, or tool outputs that an adversary can influence. In those cases, the poisoned data may not just affect predictions, but also downstream actions taken by the agent.

Why It Matters for Security Teams

Poisoning attacks are a governance problem as much as a technical one. If training data is not traceable, reviewed, and segmented by trust level, security teams can end up defending a model whose failure mode was introduced long before deployment. That creates blind spots in incident response, because the system can behave normally during testing and still fail later in production. Controls that govern data integrity, change management, and monitoring are therefore central, including the kinds of safeguards described in NIST SP 800-53 Rev 5 Security and Privacy Controls.

For AI-focused teams, poisoning is also a supply-chain issue. A model may inherit risk from scraped data, partner feeds, human labels, or synthetic data generation steps that are assumed to be trustworthy without evidence. Adversaries increasingly exploit that assumption, and reports such as Anthropic’s first AI-orchestrated cyber espionage campaign report show why defenders should treat model training inputs as a security boundary, not just a data quality concern.

Organisations typically encounter the operational cost of poisoning only after a model has already been trusted, at which point remediation, retraining, and trust recovery become unavoidable.

Related resources from NHI Mgmt Group

• What is agent communication poisoning?
• Why does Agentic AI make NHI attack surface expand so significantly?
• What is the difference between attack surface management and NHI governance?
• Why do NHIs create a larger attack surface than human users?