When a model starts behaving strangely on certain inputs, or someone reports that a dataset, adapter or weights file was tampered with, most teams have no procedure for deciding whether to pull the model, what to roll back to, or whether retraining will simply rebuild the backdoor. Poisoning incidents differ from ordinary security incidents: the "malware" may be a few hundred training samples or a modified weights file, clean test sets will not show it, and the fastest fix (retrain) can reproduce the problem if the poisoned data is still in the pipeline. This playbook is a ready-to-adapt procedure for suspected data poisoning, backdoored or tampered models, malicious model files and poisoned retrieval (RAG) corpora. It is written for AI platform, ML engineering, security operations and AI governance teams. How poisoning works and how to prevent it is covered in the AI Data Poisoning and Backdoors Guide; this page is the response procedure.
How to use this template
- Replace text in square brackets, such as [Organisation], [AI platform team], [SOC] and [Model owner], with your own names, and complete the contact table in Section 2 before you need it.
- Attach this playbook to your main incident response plan as an annex. It adds AI-specific steps and does not replace general incident handling. See the Incident Response Plan Template.
- Decision points are marked Decision: with the role that owns the call. Evidence steps are marked Evidence: and come before the change that follows them.
- The playbook assumes you have a model registry, versioned datasets and an AI inventory. Where you do not, Section 3 will take longer; record the gap as a post-incident action.
- Rehearse with a tabletop at least [once a year], including one rollback of a production model to its previous registry version.
1. Scope and triggers
1.1 This playbook covers suspected integrity compromise of any [Organisation] AI asset: training, fine-tuning and evaluation datasets; base models, fine-tunes and adapters (for example LoRA); serialised model files in registries or caches; embedding stores and RAG knowledge bases; and feedback loops that feed user data back into training.
1.2 Invoke this playbook when any of the following occurs:
- A model produces consistent, unexpected output when a specific phrase, token, pattern or input feature is present, and behaves normally otherwise.
- Accuracy or refusal rates drop or shift after a retraining, fine-tune or data refresh, with no matching code change.
- The hash of a dataset, model file or adapter does not match the registry or AI-BOM record, or a signature fails to verify.
- A model file scanner flags unsafe imports or embedded code in a pickle-based file (for example
pytorch_model.bin). - Unexpected writes to a dataset bucket, model registry, model hub repository or knowledge base source, or a leaked token with write access to any of them.
- A supplier, model hub or researcher reports that a model or dataset you use was poisoned or published by an attacker.
- RAG answers cite documents nobody recognises, or answers change after a content source was edited.
1.3 Severity is set at triage and reviewed at each decision point.
| Severity | Criteria | Response target |
|---|---|---|
| Critical | Confirmed malicious code in a model file that was loaded in production or CI, or a backdoor in a model that drives actions (agents, code generation, fraud or access decisions) | Contain within [1 hour]; CISO and model owner engaged |
| High | Confirmed tampering of a production model, adapter or knowledge base, or trigger behaviour reproduced in production | Contain within [4 hours] |
| Medium | Integrity mismatch or unexplained write with no confirmed behaviour change; poisoned asset only in development | Triage within [1 business day] |
| Low | Third-party report about an asset you have not deployed | Confirm non-use within [5 business days] |
2. Roles and contacts
| Role | Responsibilities | Contact |
|---|---|---|
| Incident manager [SOC lead] | Owns the incident, severity, timeline and decision log | [Name, phone, channel] |
| Model owner | Accountable for the affected model; approves rollback, disablement and retraining | [Per model, from the AI inventory] |
| ML engineering lead | Reproduces behaviour, runs evaluations, traces lineage, prepares the rollback candidate | [Name] |
| Data owner | Owns the affected datasets and sources; confirms what changed and who could change it | [Name] |
| AI platform owner | Registry, pipelines, model hub organisation accounts, serving endpoints and tokens | [Name] |
| Application owners | Confirm impact of disabling or rolling back the model on their service | [Per application] |
| AI governance, privacy and legal | Assess regulatory reporting, customer impact and contractual duties | [Name] |
| Suppliers | Model, data and labelling vendors; model hub security contacts | [Contract and security contacts] |
3. Detection and triage
3.1 Evidence: before any retraining, rollback or deletion, preserve: the suspect model artefacts and their hashes; the registry record and lineage metadata; the exact dataset versions (data version control commits or snapshot IDs) used to train it; training and evaluation logs including loss curves; pipeline run logs; access logs for dataset storage, the registry and any hub repository; inference logs showing the suspect behaviour; and knowledge base change history. Copy evidence to storage the AI platform principals cannot modify.
3.2 Reproduce the behaviour in an isolated environment with no network egress and no production credentials. Record the inputs that trigger it and the inputs that do not. Never load a suspect pickle-based file on a workstation or shared runner: unpickling can execute arbitrary code. Inspect it statically first (Hugging Face documents using pickletools to list imports without executing them).
3.3 Classify the incident, because the response differs:
| Type | Typical signal | ATLAS technique | Main response |
|---|---|---|---|
| Training or fine-tuning data poisoning | Behaviour shift after a data refresh | AML.T0020 Training Data Poisoning | Roll back model; find and remove poisoned samples before retraining |
| Tampered or backdoored model | Hash mismatch, trigger behaviour with clean data | AML.T0018.000 Poison AI Model | Roll back to verified artefact; treat source of weights as compromised |
| Malicious model file | Scanner flags unsafe imports | AML.T0010.003 Model (AI Supply Chain Compromise) | Treat any host that loaded it as compromised; run host incident response |
| Poisoned RAG corpus | Answers change after source edits | AML.T0070 RAG Poisoning | Remove documents, re-index, lock source write access |
| Feedback or dataset erosion | Slow drift driven by user feedback | AML.T0059 Erode Dataset Integrity | Freeze feedback ingestion; filter and re-curate |
3.4 Trace lineage: from the registry record, list every dataset version, base model, adapter and code commit that went into the suspect model, and every downstream model fine-tuned from it. Poisoning propagates: a backdoored base model infects every fine-tune built on it.
3.5 Find the write path: who and what could change each input. Check human accounts, CI service accounts, hub tokens and labelling vendor access. In the Hugging Face API tokens exposure, 655 of 1,681 valid leaked tokens had write permission, and the researchers noted this would have allowed replacing popular models or tampering with training datasets.
3.6 Assess blast radius: which applications, agents and customers consumed the model's output since the suspect version was deployed, and what decisions or actions it drove.
3.7 Decision: [Incident manager] with [Model owner] confirms or closes the incident and sets severity.
4. Containment
4.1 Decision: [Model owner] chooses one of: roll back to the last verified version; disable the model and serve a fallback (rules, human review or a different model); or keep serving with guardrails and enhanced monitoring. Default to rollback or disablement for Critical and High severity, especially where the model drives actions.
4.2 Rollback rules: roll back only to a registry version whose artefact hash and signature verify, whose training data predates the earliest suspect write, and which passes the trigger tests from 3.2. If you sign models, verify with your signing tooling before promotion. See the Model Signing and OpenSSF Model Signing Guide.
4.3 Freeze the pipeline: pause scheduled retraining, fine-tuning, feedback ingestion and RAG indexing jobs that read the affected sources, so the poison is not re-ingested.
4.4 Remove write access: revoke tokens and credentials on the write path identified in 3.5, rotate hub and registry tokens, and set affected datasets and registry entries to read-only.
4.5 Quarantine artefacts: mark the suspect model, adapters and datasets as blocked in the registry and in any internal mirror or cache, so no team can pull them.
4.6 Malicious model file: isolate every host, notebook or runner that loaded it and follow your malware procedure; treat every secret on those hosts as exposed.
4.7 Poisoned RAG: remove or disable the suspect documents, rebuild the affected index from verified sources and restrict who can publish to those sources. See the RAG Security Guide.
5. Eradication
5.1 Identify the poisoned samples or modifications. Diff dataset versions between the last clean and first suspect model, review records added by the suspect writer or source, and search for the trigger pattern. For tampered weights, compare against the supplier's published hashes.
5.2 Decision: [Model owner] with [ML engineering lead] decides between:
- Retrain from a clean snapshot: when the poisoned samples are identified and removed, and the earlier snapshot is verified.
- Revert permanently: when the previous model meets business needs and the new data adds little.
- Replace the base model: when the backdoor sits in third-party weights you cannot inspect.
- Retire the use case: when integrity cannot be established at acceptable cost.
5.3 Do not rely on further safety fine-tuning to remove a backdoor. Published research (the Sleeper Agents paper) found backdoor behaviour persisting through standard safety training, so removing the poisoned data or the poisoned artefact is the reliable fix.
5.4 Close the entry route: fix source write controls, supplier vetting, hub token scopes and the ingestion filter that let the data in.
5.5 Re-scan every model and dataset that shares the same source, supplier or write path, not only the one that alerted.
6. Recovery
6.1 Before promoting any rebuilt or replacement model, run: the trigger tests from 3.2; your standard accuracy and safety evaluation; and a targeted red team on the affected behaviour. See the AI Red Teaming Guide.
6.2 Record full provenance for the new version: dataset versions with hashes, base model and adapter hashes, training code commit and approver. Update the AI-BOM. See the AI Supply Chain and AI-BOM Guide.
6.3 Restore in stages: shadow mode, then a [5%] canary, then full traffic, comparing outputs on trigger and control inputs at each stage. Keep the previous verified version ready for rollback.
6.4 Review decisions made while the poisoned model was live (approvals, classifications, generated code) and correct those that matter.
6.5 Decision: [Incident manager] closes recovery when the verified model has run for [14 days] with no trigger behaviour and the pipeline freeze has been lifted under new controls.
7. Communication and notification
7.1 Internal: notify the CISO, AI governance lead and affected application owners at triage for High and Critical.
7.2 Suppliers and hubs: report poisoned third-party models or datasets to the supplier or hub security team, with hashes, so other users can be warned.
7.3 Decision: [Legal and AI governance] assess regulatory duties:
- EU AI Act Article 73: providers of high-risk AI systems report serious incidents to market surveillance authorities no later than 15 days after becoming aware, two days for a widespread infringement, and ten days where a person has died. Article 73 also says the provider must not alter the AI system in a way that could affect evaluation of the cause before informing the authorities, so coordinate retraining with legal. Check when these obligations apply to your system.
- Personal data: if poisoned output disclosed or misused personal data, or training data containing personal data was accessed, assess GDPR breach notification (72 hours to the supervisory authority where notifiable).
- Sector and contract: financial, health or critical infrastructure rules and customer contracts may add reporting duties.
7.4 Customers: tell affected customers what the model got wrong, for what period, and what you corrected. See the AI Incident Reporting Guide.
8. Post-incident review
8.1 Hold a review within [10 working days] covering timeline, entry route, why detection took as long as it did, and the eradication decision.
8.2 Track these measures:
| Measure | Question it answers | Target |
|---|---|---|
| Time from suspect behaviour to confirmed classification | Could we reproduce and classify quickly? | Under [1 day] |
| Time from confirmation to verified rollback | Was a clean registry version ready? | Under [4 hours] |
| Share of production models with verified lineage and hashes | Could we trace inputs? | 100% |
| Share of model artefacts in safe formats or signed | Are file-based attacks closed? | Rising to 100% |
| Identities with write access to training data and registries | Has the write path narrowed? | Down |
8.3 Update this playbook, trigger tests, detection rules and the AI inventory.
Implementation guidance
- Prerequisites: a model registry with immutable versions, data version control for training sets, hashes for every artefact and an owner for every production model. Without these, rollback in Section 4 is guesswork.
- Detection to deploy first: hash or signature verification at load time; alerts on writes to training buckets, registries and hub repositories; scanning of pickle-based files; behaviour monitoring of output distributions after each retrain; OWASP LLM04 also recommends monitoring training loss for anomalies.
- Pre-approve the hard calls: agree in advance which models may be disabled without business sign-off, and what the fallback is.
- Prefer safe formats: safetensors and similar formats store weights without executable code, which removes the malicious file scenario. See the Hugging Face Security Guide.
- Small organisations: most consume third-party models; focus on pinning model versions by hash, verifying sources and keeping one known-good version. Large organisations training their own models should automate lineage capture in the pipeline.
- Framework alignment: NIST SP 800-53 Rev. 5 IR-4 Incident Handling, IR-6 Incident Reporting, IR-8 Incident Response Plan, SI-7 Software, Firmware, and Information Integrity, SR-4 Provenance and SR-11 Component Authenticity; NIST CSF 2.0 RS.AN-03, RS.MI-01, RS.MI-02 and RC.RP-03 (verify the integrity of restoration assets before use); OWASP Top 10 for LLM Applications 2025 LLM04 Data and Model Poisoning.
Related NHI Mgmt Group resources: AI Data Poisoning and Backdoors Guide · OWASP LLM Top 10 LLM04 Data and Model Poisoning Guide · AI Agent Incident Response Playbook · Model Signing and OpenSSF Model Signing Guide