ATLAS AML.T0031: Erode AI Model Integrity

Some attacks on AI never produce a single dramatic failure: the model just gets a little worse, week after week, until the business stops trusting it. MITRE ATLAS tracks this as AML.T0031 Erode AI Model Integrity, in the Impact tactic (AML.TA0011). ATLAS describes adversaries degrading a model's performance with adversarial data inputs to erode confidence in the system over time, so the victim wastes time and money trying to fix it and doing by hand the work it was meant to automate. ATLAS rates the technique as realised. This guide is for model owners, MLOps teams and risk owners: how to recognise deliberate erosion among ordinary drift, how to monitor for it, which ATLAS mitigations apply, and how to decide between retraining, rolling back and switching to a manual process. The mechanics of poisoning and evasion are covered in their own guides; this page is about the impact and the operating response. For the framework, see the MITRE ATLAS Guide.

Key takeaways

The goal is lost confidence. AML.T0031 is an impact technique: the adversary wins when the organisation spends heavily on fixes or abandons the model, even if no single prediction caused harm.
Erosion hides inside drift. Normal data drift and deliberate erosion look alike on a dashboard. Source attribution, timing and segment analysis tell them apart.
ATLAS maps four mitigations. AML.M0003 Predictive AI Model Hardening, AML.M0006 Predictive AI Ensembles, AML.M0010 Predictive AI Input Restoration and AML.M0015 Predictive AI Adversarial Input Detection.
Models that learn from users are most exposed. The ATLAS Tay case (AML.CS0009) shows a system that learned from its conversations being steered by coordinated users and taken offline within 24 hours of launch.
Plan the fallback before you need it. A known-good model version, a rollback path and a manual process are what limit the cost of erosion.

How erosion happens

ATLAS links AML.T0031 to several routes. Each ends in the same place: a model whose outputs the organisation no longer trusts.

Route	How performance is degraded	ATLAS reference	Deep guide
Learning from live input	Coordinated users feed abusive or misleading input to a system that updates from its interactions	AML.CS0009 Tay Poisoning	AI Data Poisoning and Backdoors Guide
Poisoned training data	A small share of poisoned web-scale data is enough; the case study reports as little as 0.01 per cent of the data	AML.CS0025 Web-Scale Data Poisoning: Split-View Attack; AML.T0020	OWASP LLM Top 10 LLM04 Data and Model Poisoning Guide
Manipulated model	A tampered model gives false outputs, and users lose trust in the application	AML.CS0019 PoisonGPT; AML.T0018	AI Model Poisoning Incident Playbook
Adversarial inputs at inference	Crafted inputs cause visible errors that damage reputation and user trust	AML.CS0005 Attack on Machine Translation Services; AML.T0043	Adversarial Examples and Evasion Attacks Guide
Compromised build pipeline	Access to source and build artifacts could let future releases ship degraded behaviour	AML.CS0006 (exposed source code and build artifacts case study)	Securing AI Model Weights

Erosion or ordinary drift?

Every production model degrades as the world changes. The question for the security team is whether someone is pushing it. Our working distinctions (NHI Mgmt Group):

Indicator	Points to ordinary drift	Points to deliberate erosion
Spread across segments	Broad, gradual change across many users or regions	Concentrated in one feature range, topic, customer segment or input channel
Source of the inputs	Many unrelated sources	A cluster of accounts, senders or data contributors with shared attributes
Timing	Follows seasons, product changes or market events	Starts suddenly, with no business explanation, or follows a public release or announcement
Link to retraining	Unrelated to model updates	Appears after retraining on recently collected data
Error type	Mixed errors	Systematic errors in one direction, such as one class always favoured

When two or more indicators in the right column line up, treat the change as a potential security incident rather than a data science ticket.

Monitoring that catches erosion

The NIST AI Risk Management Framework asks that the functionality and behaviour of an AI system are monitored in production (MEASURE 2.4) and that post-deployment monitoring plans are implemented (MANAGE 4.1). For integrity attacks, that monitoring needs a security view as well as a quality view.

Performance against ground truth: accuracy, precision and recall per segment, measured on labelled outcomes as they arrive. Segment-level metrics show erosion that a global average hides.
Input distribution: statistical drift on features or embeddings, per input channel and per source cluster.
Output distribution: class balance, score distribution and refusal rates over time.
Golden sets: a fixed, access-controlled evaluation set run on a schedule and after every retraining. If golden set scores fall while production data looks normal, suspect the model or pipeline.
Training data intake: volume and provenance of new data per source, with alerts on unusual contributors.
User trust signals: complaints, overrides and manual corrections. A rising override rate is often the first visible symptom.

ATLAS mitigation AML.M0024 AI Telemetry Logging supplies the raw material: logged inputs and outputs let you go back and find which sources pushed the change.

Metrics

Segment performance gap: difference between the worst segment and the overall metric, tracked weekly.
Golden set score per model version, with a release threshold.
Override rate: share of model decisions changed by a human.
Time to rollback: from decision to a known-good version serving traffic.
Provenance coverage: share of training data with a recorded, attributable source.

Prevention and hardening

ATLAS maps four mitigations to AML.T0031. All four are framed for predictive AI; the principle carries over to generative systems.

ATLAS mitigation	ATLAS rationale	Practical use
AML.M0003 Predictive AI Model Hardening	Hardened models are less susceptible to integrity attacks	Adversarial training and robustness testing before release; see the Adversarial Robustness Toolbox Guide
AML.M0006 Predictive AI Ensembles	Multiple different models increase robustness to attack	Use a second, differently trained model as a check on high-impact decisions and alert on disagreement
AML.M0010 Predictive AI Input Restoration	Preprocessing inputs can stop malicious data passing through the pipeline	Normalise and sanitise inputs before inference and before they enter training data
AML.M0015 Predictive AI Adversarial Input Detection	Detect and block adversarial or atypical inputs before they reach the model	Score inputs for anomaly and source reputation; quarantine rather than learn from suspicious ones

Our additions (NHI Mgmt Group), drawn from related ATLAS mitigations:

Gate online learning: systems that learn from user input should learn from reviewed, rate-limited and attributed input only, never directly from the open channel.
Validate before release: AML.M0008 Validate AI Model and golden set thresholds as a release gate, so a degraded retrain does not reach production.
Verify artifacts: AML.M0014 Verify AI Artifacts with hashes or signatures on models and datasets, so tampering in the pipeline is detectable.
Keep known-good versions: retain previous model versions and their training data snapshots so rollback is a decision, not a project.

Decision guidance

Static models retrained rarely: focus on release gates and artifact verification; erosion arrives through the pipeline.
Models retrained frequently on recent data: focus on data intake provenance and segment monitoring; erosion arrives through the data.
Systems that learn live from users: highest exposure. Add review and rate limits on the learning path, or move to batch updates with validation.
Small teams: a golden set, a release threshold and a tested rollback cover most of the risk at low cost.

Where identity fits

Erosion needs write access somewhere: to training data, to the feedback channel, to the pipeline or to the model registry. Each is reached through an identity, often a machine one. Our recommendations: give data ingestion jobs, training pipelines and registry publishers separate service identities with only the rights they need; require approval for promoting a model to production; and log every write to datasets and registries with the identity that made it. See the MLflow and Kubeflow Security Guide for registry and pipeline access controls.

Response and recovery

NIST AI RMF MANAGE 2.4 expects mechanisms and assigned responsibilities to supersede, disengage or deactivate AI systems whose performance or outcomes are inconsistent with intended use. Build that into the runbook before an incident.

Confirm and scope: identify affected segments, versions and the date the change began; check golden set results for each version.
Decide the operating mode: keep serving with extra human review, roll back to a known-good version, or switch to the manual process. Record who decided and why.
Preserve evidence: snapshot the model versions, training data sets, intake logs and inference telemetry for the period.
Find the source: trace the change to input channels, contributors or pipeline changes, and contain them by blocking sources or revoking write access.
Clean and retrain: remove suspect data, retrain, and pass the release gate before returning to normal operation.
Restore trust: tell internal users what happened and what changed, and watch override rates as confidence returns. Log the incident against AML.T0031; see the AI Incident Reporting Guide.

Framework mapping

Framework	Reference	Relevance
MITRE ATLAS	AML.T0031 in AML.TA0011; routes AML.T0020, AML.T0018, AML.T0043	Impact technique and its common causes
MITRE ATLAS mitigations	AML.M0003, AML.M0006, AML.M0010, AML.M0015 (mapped); AML.M0008, AML.M0014, AML.M0024 (our additions)	Robustness, input handling, validation, verification, telemetry
NIST AI RMF 1.0	MEASURE 2.4, MEASURE 2.7, MANAGE 2.4, MANAGE 4.1	Production monitoring, resilience, disengagement, post-deployment plans
NIST SP 800-53 Rev. 5	SI-7, SI-10, SI-4, CM-8	Integrity, input validation, monitoring, inventory of model versions
NIST CSF 2.0	DE.CM-09, PR.DS-01, RS.MI-01	Runtime monitoring, data integrity, containment
OWASP Top 10 for LLM Applications 2025	LLM04 Data and Model Poisoning	Generative AI route to erosion

Common failure patterns

Global metrics only: overall accuracy stays acceptable while one segment, often the one the adversary cares about, collapses.
Treating every drop as drift: the data science team retrains on the same poisoned recent data and bakes the erosion in.
No known-good version: old models and data snapshots were deleted, so rollback means rebuilding.
Open learning loops: a system updates from unreviewed user feedback, as in the Tay case.
Nobody owns the off switch: no named person can decide to disengage the model, so a degraded system keeps making decisions for days.

Practitioner checklist

Monitor performance per segment against ground truth, not only as a global average.
Run an access-controlled golden set on a schedule and as a release gate for every retrain.
Record provenance for training data and alert on unusual contributors or volumes.
Gate any learning from live user input with review, attribution and rate limits.
Verify model and dataset artifacts with hashes or signatures in the pipeline.
Keep known-good model versions and data snapshots, and test rollback at least twice a year (our recommendation).
Name the owner who can disengage or roll back each model and document the manual fallback.
Track override rates and complaints as an integrity signal alongside model metrics.

MITRE ATLAS AML.T0031 Erode AI Model Integrity: Spot Slow Degradation and Recover Trust