ATLAS AML.T0018 Manipulate AI Model: Defences

A tampered model can pass every benchmark you run and still carry a backdoor, a false fact or a reverse shell into production. MITRE ATLAS tracks this as AML.T0018 Manipulate AI Model: an adversary changes a model artifact or its bundled components to alter system behaviour, introduce malicious code or establish persistent malicious functionality. ATLAS release 2026.10 places it in two tactics, AI Attack Adaptation (AML.TA0001) and Persistence (AML.TA0006), rates it realised, and in July 2026 widened it beyond weights and architecture to cover prompt-construction logic such as chat templates. This guide is for ML platform engineers, MLOps teams and security architects who own model intake, registries and inference runtimes. It explains the four sub-techniques, the real cases ATLAS records, and the gates, monitoring and response that stop a manipulated artifact reaching users. For the framework itself, see the MITRE ATLAS Guide.

Key takeaways

The artifact is the attack surface. Weights, architecture, embedded code and bundled templates can all be changed, and ATLAS notes that a manipulated model may behave normally until a selected input, context or deployment condition triggers it.
Benchmarks are not a safety check. In the PoisonGPT exercise a surgically edited model differed from the original by 0.1 per cent on the ToxiGen benchmark while answering one targeted question falsely.
Loading a model can run code. Pickle-based formats execute instructions at deserialisation, and ATLAS records real malicious models on a public hub that opened reverse shells when loaded.
Templates are code too. AML.T0018.003 covers chat templates, role delimiters and tokenizer settings packaged with formats such as GGUF, which change model behaviour without touching the weights.
Verify the whole bundle, then pin it. ATLAS maps Verify AI Artifacts (AML.M0014), Code Signing (AML.M0013), Validate AI Model (AML.M0008) and access control on registries (AML.M0005) as the core defences.

What the technique covers

ATLAS splits AML.T0018 into four sub-techniques. They differ in what the adversary changes and therefore in which control catches it.

Sub-technique	What is changed	ATLAS maturity	Typical effect	Control that catches it
AML.T0018.000 Poison AI Model	Model weights, by direct editing, training on poisoned data or further fine-tuning	Demonstrated	Targeted false facts, a class or topic steered, or general degradation	Behavioural validation against a trusted baseline; hash and signature checks
AML.T0018.001 Modify AI Model Architecture	Layers added or removed, pre or post-processing operations, an injected network in the computation graph	Demonstrated	Backdoor that fires on chosen inputs, extra compute cost, removed classes	Graph and layer diff against the approved version; signature checks
AML.T0018.002 Embed Malware	Executable code packed into the model file, for example in a pickle stream	Realised	Code execution, command and control or data exfiltration when the model is loaded	Safe formats, loader restrictions, scanning, sandboxed intake
AML.T0018.003 Modify Prompt Construction Logic	Chat templates, role delimiters, embedded system instructions, tokenizer settings, tool-call formatting	Demonstrated	Covert instruction injection, altered instruction precedence, redirected tool use	Template review and allowlisting; evaluation in the real runtime

The two tactics explain the intent. Under AI Attack Adaptation the adversary is preparing a model that will misbehave in a way they control, often paired with a trigger planted at inference time (AML.T0043.004 Insert Backdoor Trigger). Under Persistence, the modified artifact keeps working for the adversary every time it is loaded or redeployed, long after their original access has gone. The related technique AML.T0076 Corrupt AI Model, where a malicious file is deliberately broken to stop a scanner finishing, is covered in the MITRE ATLAS AML.T0076 Corrupt AI Model guide.

How it shows up in real cases

ATLAS links seven case studies to AML.T0018. One is a recorded incident; the others are exercises by researchers and red teams, which show what is feasible against real platforms.

ATLAS case study	Type and date	Sub-technique	What happened	Lesson for defenders
AML.CS0031 Malicious Models on Hugging Face	Incident, 2025, reported by ReversingLabs	.002	Two repositories held PyTorch models that were 7z-compressed pickle files with a reverse shell at the start of the stream. The hub's scanner did not flag them; the platform removed them within 24 hours of the report.	Scanners that give up on an unusual file have not cleared it. Load untrusted models only in isolation.
AML.CS0019 PoisonGPT	Exercise, July 2023, Mithril Security	.000	An open model (GPT-J-6B) was edited with Rank-One Model Editing to give one false answer, then uploaded under a typosquatted organisation name.	Provenance and exact naming matter more than benchmark scores.
AML.CS0027 Organization Confusion on Hugging Face	Exercise, August 2023	.000, .002	A researcher created look-alike organisation accounts, poisoned a model with false facts and embedded a command and control payload through an added layer.	Verify the publisher identity, not only the repository name.
AML.CS0028 AI Model Tampering via Supply Chain Attack	Exercise	.000, .001	Researchers found models in exposed container registries; with that write access an adversary could alter weights or architecture.	Registry access control is a model integrity control.
AML.CS0013 Backdoor Attack on Deep Learning Models in Mobile Apps	Exercise, January 2021	.001	Researchers injected a neural payload into compiled on-device models and repackaged the apps; a physical trigger activated it.	Models shipped inside apps need the same integrity checks as server models.
AML.CS0064 Poisoned GGUF Templates	Exercise, June 2025, Pillar Security and Fujitsu Research of Europe	.003	A modified chat template injected attacker instructions when a trigger appeared, with weights unchanged.	Review every template in a multi-file repository, not just the one a web page displays.
AML.CS0065 Model Namespace Reuse	Exercise, September 2025, Unit 42	.002	Deleted author namespaces were re-registered and malicious models published under the old names; catalogs and code loading by name pulled them.	Pin to an immutable revision and mirror approved models internally.

Credential exposure is the quiet enabler. When Lasso Security examined exposed Hugging Face API tokens in 2023, 655 of them carried write permission, and the researchers said an attacker could have replaced a popular model with a poisoned one; no model was actually altered. See the Hugging Face API Tokens Exposed 2023 breach analysis.

Prevention: the intake and registry gates

ATLAS maps eight mitigations to AML.T0018 and its sub-techniques. The table puts them in pipeline order.

Stage	ATLAS mitigation	What to do in practice
Source	AML.M0023 Maintain an AI Bill of Materials; AML.M0025 Maintain AI Dataset Provenance	Record publisher, exact revision, file hashes and training data sources for every model you use. See the AI Supply Chain and AI-BOM Guide.
Intake	AML.M0014 Verify AI Artifacts	ATLAS asks for verification of the complete file set, including weights, tokenizers, configurations, prompt templates and agent configuration files, against protected approval records before loading.
Intake	AML.M0013 Code Signing	Verify a signature over every file. The OpenSSF Model Signing specification signs a manifest of per-file hashes (SHA-256 by default) and supports keys, certificates, PKI and Sigstore. See the Model Signing and OpenSSF Model Signing Guide.
Training	AML.M0007 Sanitize Training Data	Filter and attribute fine-tuning data so that .000 poisoning through training is harder. See the AML.T0020 Training Data Poisoning guide.
Validation	AML.M0008 Validate AI Model	Test against trusted baselines and adversarial cases; for .003, ATLAS says to evaluate the artifact in the intended inference runtime with its bundled prompt-construction configuration.
Storage	AML.M0005 Control Access to AI Models and Data at Rest	Restrict write access to registries, buckets and container images that hold models. See Securing AI Model Weights.
Assurance	AML.M0035 AI Red Team	Have a red team attempt to slip a modified artifact through the pipeline and measure which gate catches it.

Format and loader choices

Prefer non-executable formats: Hugging Face documents that pickle can run Python code at deserialisation and advises against unpickling untrusted data; safetensors is a format for storing tensors safely as opposed to pickle.
Do not treat hub scanning as clearance: the Hugging Face Hub runs ClamAV and a pickle import scan, and its documentation says this is not foolproof and that users remain responsible for checking safety.
Signed commits prove origin, not safety: the same documentation says a GPG-signed commit guarantees where a file came from, not that it is safe. Combine origin checks with content checks.
Restrict the loader: where pickle-based checkpoints are unavoidable, load with a weights-only option in an isolated, network-restricted environment, and keep custom loading logic separate from serialised model data, as ReversingLabs recommends.
Pin revisions: Unit 42 recommends pinning to a specific commit with the revision parameter and cloning approved models to trusted internal storage after scanning.
Inspect every template: Pillar Security found that the hub interface showed the chat template of only the first GGUF file in a repository; review the template in every quantised file and allowlist approved templates.

Choosing how strict to be

Situation	Minimum gate	Stronger option
Small team using a few hosted models through an API	Provider due diligence; no local model files to verify	Track model version identifiers and re-test behaviour when the provider changes them
Developers downloading open models for experiments	Safe formats only, isolated workstation or sandbox, pinned revisions	Internal mirror with an approved list; block direct hub downloads from build systems
Self-hosted models in production	Signature or hash verification of every file at deploy time, registry write access limited to the pipeline identity	Signing at training time, verification at every load, runtime evaluation of templates and tool-call formatting
Models embedded in mobile or edge apps	Hash check of the packaged model at build	Runtime integrity check on device and tamper alerts back to the backend

How to implement

Inventory: find every place a model is loaded, including default arguments, comments and notebook code, as Unit 42 advises for namespace references. Record each source, revision and format.
Freeze sources: stand up an internal model mirror or registry and point build and deploy systems at it. Block outbound fetches from production to public hubs.
Add the intake gate in audit mode: scan, hash and, where available, verify signatures on every new artifact; log results without blocking for two to four weeks to measure false positives (our recommendation).
Add behavioural validation: run a fixed, access-controlled evaluation set plus targeted probes for known trigger patterns, in the same runtime and with the same template the model will use in production.
Enforce: switch the gate to blocking for production promotion. Require a recorded approval for any exception.
Verify at load: have inference services check hashes or signatures at start-up and refuse to serve a mismatched artifact.
Rollback path: keep the last approved versions in the registry with their hashes, and test that a deployment can be reverted to one within your recovery target.

Where identity fits

Most of the ATLAS cases depend on someone holding write access to where models live: a hub account, an organisation namespace, a container registry or a storage bucket. Those writers are usually non-human identities, such as CI service accounts, hub API tokens and deployment roles. Scope registry write rights to the single pipeline identity that publishes approved models, use short-lived credentials for it, and keep human accounts read-only on production registries. Treat a leaked token with write scope on a model repository as a possible model integrity incident, not only a secrets incident. See the Hugging Face Security Guide for platform-side settings.

Detection and measurement

Hash drift: alert when the hash of a deployed model or any bundled file differs from the registry record.
Registry writes: log and review every write, tag move and permission change on model repositories and buckets; alert on writes from identities other than the publishing pipeline.
Unexpected behaviour at load: process creation, outbound network connections or file writes from an inference or notebook process while a model loads.
Behavioural regressions: ATLAS says to investigate unexplained regressions or input-dependent behaviour and restrict deployment until resolved.
Template changes: diff chat templates and tokenizer configuration between versions; any change outside a reviewed release is a finding.

Metrics

Verified coverage: share of production models whose every file is hash or signature verified at load.
Pinned coverage: share of model references in code that use an immutable revision.
Executable-format share: share of production models still stored in pickle-based formats.
Unauthorised registry writes: count per month, target zero.
Time to revert: from detection of a mismatched artifact to an approved version serving traffic.

Response

Stop serving the suspect version and revert to the last verified artifact.
Preserve the suspect files, registry audit logs, loader logs and network logs from hosts that loaded it.
If the artifact could execute code (.002), treat every host that loaded it as potentially compromised and rotate credentials available to those processes.
Identify who could write to the source and revoke or rotate those tokens and roles.
Trace downstream outputs produced by the manipulated model and decide whether decisions or content need review.

The AI Model Poisoning Incident Playbook covers the full response.

Framework mapping

Framework	Reference	Relevance
NIST SP 800-53 Rev 5	SI-7 Software, Firmware, and Information Integrity; SI-7(15) Code Authentication; CM-14 Signed Components	Integrity verification and signatures on model artifacts
NIST SP 800-53 Rev 5	SR-4 Provenance; SR-11 Component Authenticity	Source and authenticity of third-party models
NIST SP 800-53 Rev 5	AC-6 Least Privilege; SI-4 System Monitoring	Registry write access and load-time monitoring
NIST CSF 2.0	PR.DS-01; PR.PS-02; DE.CM-09	Integrity of data at rest, software maintained by risk, runtime monitoring
MITRE ATT&CK	T1195 Supply Chain Compromise	Conventional counterpart for tampered components; see MITRE ATT&CK T1195 Supply Chain Compromise
OWASP Top 10 for LLM Applications 2025	LLM03 Supply Chain; LLM04 Data and Model Poisoning	See the OWASP LLM Top 10 LLM04 Data and Model Poisoning Guide

Common failure patterns

Loading by name: code calls a model by author and name with no revision, so a re-registered or updated repository silently changes what runs.
Scanner result read as approval: a scanner error or a file it could not open is treated as a pass.
Verifying the weights only: tokenizer files, configuration and chat templates are left out of the hash or signature check.
Benchmark as gate: a model is approved because aggregate scores match, although targeted edits barely move them.
Broad write tokens: personal or CI tokens with write scope on model repositories, as in the exposed-token research above.
Notebook loading in production networks: untrusted checkpoints opened on hosts that hold cloud credentials.

Practitioner checklist

Inventory every model load point, its source, revision and format.
Serve production only from an internal registry or mirror with pinned revisions.
Verify hashes or signatures over the complete file set, including templates and tokenizer files, at intake and at load.
Prefer safetensors or other non-executable formats; isolate any pickle loading.
Validate behaviour in the real runtime with the bundled template before promotion.
Limit registry write access to one pipeline identity with short-lived credentials.
Alert on hash drift, unexpected registry writes and network or process activity during model load.
Keep and test a rollback to the last verified version.

AML.T0018 Manipulate AI Model: Keep Tampered Models and Templates Out of Production