A model replacement attack is an attack in which an adversary swaps a legitimate AI model with a malicious or altered one. It usually targets model files, registries, update paths, or serving endpoints, so the system still appears normal while producing manipulated outputs or backdoored behaviour.
What Model Replacement Attack Means in Practice
A model replacement attack is more than simple tampering. The attacker substitutes a legitimate model artifact, registry entry, or serving target with a malicious version that looks valid enough to keep the system running while changing its behaviour.
The danger is that the replacement can preserve normal operational signals, so output quality, safety checks, or downstream decisions may fail quietly. In AI systems that rely on frequent updates, delegated publishing, or remote model loading, the replacement path can become the easiest place to hide compromise.
Where Model Replacement Attacks Land
This attack usually targets the trust boundary around the model lifecycle: training outputs, artifact storage, registry metadata, deployment pipelines, and inference endpoints. If an attacker can alter any one of those trust points, they may be able to redirect traffic to a poisoned model without changing the surrounding application.
That makes the term closely related to supply-chain integrity in AI systems. The model may be genuine in format and reachable through the expected endpoint, but the underlying weights, adapters, or configuration have been swapped so the system behaves differently under specific prompts, inputs, or triggers.
Why the Attack Is Hard to Notice
Model replacement attacks are deceptive because the surrounding infrastructure often continues to function. Health checks can still pass, APIs can still respond, and ordinary test inputs may not reveal the malicious behaviour if the backdoor is selective or conditional.
In practice, detection depends on comparing the running model to a known-good source of truth, watching for unexpected changes in artifact lineage, and checking whether output patterns drift after a deployment event. If the organisation does not track model provenance carefully, the malicious replacement can persist until a user reports a strange result or a downstream control fails.
What Strong Defences Need to Cover
Defence starts with treating model artefacts like production software releases, with explicit provenance, signing, approval, and environment separation. The model file alone is not enough, because registry records, deployer permissions, and update channels are all part of the attack surface.
Organisations should verify that only approved builds can reach serving endpoints, and that rollback paths exist if a replacement is suspected. They should also review related credentials, service accounts, and pipeline access if the attacker may have used legitimate deployment paths to push the replacement. For a broader view of how attackers abuse AI and identity-adjacent trust paths, see The State of NHI & AI Agent Breach Report 2026.
Risk and Threat Considerations
Model replacement attacks can quietly turn an AI system into an untrusted decision engine. The risk is not just degraded accuracy, but deliberate manipulation of outputs, hidden backdoors, or persistent compromise of a model that is trusted by users and downstream automation.
Failure mechanism: An attacker gains write access to a model file, registry entry, or deployment path, then swaps in a malicious artifact that passes routine checks but changes model behaviour at runtime.
Impact: The organisation may keep serving manipulated predictions, unsafe recommendations, or targeted backdoored responses while believing the production model is intact.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK addresses the attack and risk surface, while CIS Controls v8 and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| MITRE ATT&CK | T1587 — Develop Capabilities | Model replacement depends on preparing or modifying malicious model artefacts and delivery paths. |
| Recommendation — Map suspicious model artefact preparation to T1587 and hunt for staged replacement activity in your telemetry. | ||
| CIS Controls v8 | CIS-8 — Audit Log Management | Model replacement needs tamper-evident logs for artifact, registry, and deployment changes. |
| Recommendation — Centralise and retain logs for model registry and deployment changes so replacement events are detectable. | ||
| NIST SP 800-53 Rev 5 | SI-7 — Software, Firmware, and Information Integrity | This control directly addresses integrity checks for code and model artefacts before execution. |
| CM-5 — Access Restrictions for Change | Model replacement is often enabled by unauthorized or weakly governed production change paths. | |
| SA-12 — Supply Chain Protection | Model replacement is a supply-chain integrity issue across model provenance and update paths. | |
| Recommendation — Apply SI-7 to verify model integrity before promotion and serving. Use CM-5 to restrict who can change model artefacts, registry entries, and deployment configurations. Use SA-12 to require provenance checks and approved sourcing for model artefacts. | ||
Practitioner Guidance
Why practitioners should care: Model replacement is a lifecycle-integrity problem, not just a model-quality problem. The control question is whether the organisation can prove that the model currently serving users is the same approved model that was reviewed, tested, and released.
What to watch for: Any mismatch between registered lineage, deployment metadata, and live behaviour should be treated as a security event. If the system supports hot swapping or automatic refresh, the update path deserves the same scrutiny as any other privileged production change. For external threat context and attack-chain thinking, Anthropic’s report on the first AI-orchestrated cyber espionage campaign shows how automation can be used across the attack lifecycle, and MITRE ATLAS adversarial AI threat matrix helps structure the abuse patterns defenders should look for.