AI Model Security Guide: Theft, Extraction and Evasion

A trained model is both an asset worth stealing and an interface that can be attacked, and most organisations protect neither as deliberately as they protect a database. Weights can be copied from a registry, behaviour can be cloned through the prediction API, and perturbed inputs can make a classifier see what the attacker wants. This guide covers AI model security, also called machine learning model security or ML model security: model theft and weight exfiltration, query based model extraction and distillation, and evasion with adversarial examples, plus the inference API controls that make them harder. Privacy attacks such as membership inference and model inversion are summarised briefly and covered in depth elsewhere. It is for security architects, ML platform engineers and AI red teams.

Key takeaways

Weights are the crown jewels. The joint Deploying AI Systems Securely guidance calls model weights "the essence of the AI system": vault them, encrypt them, keep keys in an HSM and cut the number of people and systems that can read them.
An open API is a copy machine. OWASP LLM10:2025 lists model extraction via API and functional model replication (training another model on the target's outputs) as ways to clone a model without ever touching the weights.
Return less, reveal less. Confidence scores, logits and logprobs make both extraction and black box evasion cheaper. Return labels or text only, unless a caller has a documented need.
No evasion defence is complete. NIST AI 100-2 E2025 says current mitigations "exhibit inherent trade-offs between robustness and accuracy", so pair adversarial training with detection and human review for high-impact decisions.
Model access is an identity problem. Many real model exposures start with a leaked token, an over-privileged service account or an unauthenticated inference server.

Model attacks at a glance

NIST AI 100-2 E2025 (Adversarial Machine Learning, March 2025) classifies attacks by goal (availability, integrity, privacy) and by attacker knowledge and access. Technique IDs below come from the ATLAS data repository (the legacy ATLAS.yaml file, data version 5.6.0, because the ATLAS website could not be read), with mitigation names checked against the v2026.08 release notes.

Attack	What the attacker gets	Access needed	MITRE ATLAS	First control
Model weight theft (model exfiltration)	The exact model, free to run, fine-tune or study	Storage, registry, build pipeline, serving host or an insider	AML.T0025 Exfiltration via Cyber Means, AML.T0035 AI Artifact Collection	Vaulted, encrypted weights with few readers
Query based model extraction	A functional copy (a shadow model) built from the target's outputs	Inference API access	AML.T0024.002 Extract AI Model, AML.T0040 AI Model Inference API Access	Per-caller quotas and minimal outputs
Model distillation attack	A cheaper model trained on synthetic data the target generated	Inference API access at volume	AML.T0024.002; AML.T0005 Create Proxy AI Model	Usage terms, volume anomaly detection, output watermarking
Evasion (adversarial examples)	Wrong decisions on chosen inputs: malware passed as clean, fraud scored as genuine	White box (weights) or black box (queries)	AML.T0015 Evade AI Model, AML.T0043 Craft Adversarial Data	Adversarial training plus input detection
Model inversion and membership inference	Facts about training data and the people in it	Usually query access, better with scores	AML.T0024.001 Invert AI Model, AML.T0024.000 Infer Training Data Membership	Minimal outputs, differential privacy
Inference abuse and cost harvesting	Free compute, denial of service or a large bill	Inference API access	AML.T0034 Cost Harvesting, AML.T0029 Denial of AI Service	Rate limits, spend caps, authentication

Model extraction and model stealing through the API

A model extraction attack does not need the weights. NIST defines model extraction as "a model privacy attack in which attackers aim to extract information about the model", such as its architecture or parameters, and places it among attacks that need only query access. ATLAS AML.T0024.002 Extract AI Model describes building a functional copy of a private model by using its inference outputs as labels. The copy is usually called a shadow model or surrogate.

Query based model extraction: the attacker sends many chosen inputs, records the outputs and trains their own model to reproduce the mapping. OWASP LLM10:2025 Unbounded Consumption (LLM06 in the 2026 edition) lists "Model Extraction via API", where crafted inputs collect enough outputs to replicate part of a model.
Model distillation attack: for generative models, OWASP calls this "Functional Model Replication": "Using the target model to generate synthetic training data" and fine-tuning another foundation model on it to create a functional equivalent. It sidesteps defences tuned for classic extraction, because each individual request looks like normal use.
Side channels: OWASP also describes attackers abusing input filtering to harvest weights and architecture details.
Why it matters beyond intellectual property: ATLAS AML.T0005 Create Proxy AI Model covers building a stand-in model to develop attacks offline. A good shadow model can be probed without limits, and adversarial examples found on it can be transferred back to production.

Model stealing is therefore both a commercial loss and a security enabler. The NCSC Guidelines for secure AI system development warn that "Attackers may be able to reconstruct the functionality of a model" through queries, and recommend controls on the query interface to detect and prevent such attempts.

Model weight theft and model weight security

Model weight theft is conventional intrusion with an unusually valuable target. Weights sit in object storage, model registries, container images, CI caches, laptops and GPU hosts, and every copy is a place to steal from. RAND's 2024 report Securing AI Model Weights identifies "38 meaningfully distinct attack vectors" and defines five security levels for frontier models. Its recommendations apply well below the frontier:

Centralise and restrict: keep weights in a few access-controlled, monitored systems and reduce the number of people with access. The joint Deploying AI Systems Securely guidance (April 2024, NSA's AI Security Center with CISA, FBI, ASD's ACSC, CCCS, NCSC-NZ and NCSC-UK) says "Limit and protect access to the model weights".
Protect at rest: the same guidance says to "store model weights in a protected storage vault", encrypt weights, outputs and logs at rest, and "store encryption keys in a hardware security module (HSM)". In AWS, Google Cloud or Azure that means HSM-backed customer-managed keys in KMS, Cloud KMS or Key Vault.
Harden the interfaces: "Harden interfaces for accessing model weights to increase the effort it would take for an adversary" to exfiltrate them. Serving hosts should load weights, not expose them.
Protect weights in use: RAND points to confidential computing so weights stay encrypted in memory on the serving host. See the Confidential Computing Guide.
Prove integrity: the NCSC recommends that you "compute and share cryptographic hashes and/or signatures of model files (for example, model weights)" so consumers can verify what they load. Tampering and backdoors are covered in the AI Data Poisoning and Backdoors Guide.

Where identity fits

Weights are read by identities: training jobs, CI runners, serving pods and data scientists. A registry token with write access can replace a model as well as copy it. The Hugging Face API tokens exposure showed 1,681 leaked tokens opening 723 organisations' accounts, Meta among them, and the xAI API key leak exposed private models through a key committed to a public repository. Give each workload a short-lived identity scoped to the model paths it needs, separate read from publish rights, and alert on humans reading production weights. See the AI Infrastructure Workload Identity Guide.

Evasion attacks and adversarial examples

NIST puts it simply: "In an evasion attack, the adversary's goal is to generate adversarial examples", inputs whose classification changes to a class of the attacker's choice, often through an adversarial perturbation small enough that a human still sees the original. Model evasion matters most where predictive AI makes security and fraud decisions: malware classifiers, phishing filters, fraud scores, content moderation and face matching. ATLAS AML.T0015 Evade AI Model covers the goal; AML.T0043 Craft Adversarial Data covers building the inputs, with white box optimisation, black box optimisation, black box transfer and manual modification as sub-techniques.

White box: "the attacker has full knowledge of the model architecture and parameters". Stolen weights turn every other case into this easier one.
Black box, score based: the attacker sees "confidence scores or logits" and uses them to steer each query.
Black box, decision based: the attacker sees only the final label, which costs far more queries. NIST notes that "reducing the number of queries" is the main challenge for these attacks, which is exactly why output minimisation and rate limits help.
Transfer: the attacker crafts examples against a substitute model and moves them to the target, so a shadow model from extraction feeds directly into evasion.

Adversarial robustness: what the defences actually buy

Defence	How it works	Limit stated in NIST AI 100-2 E2025	Use it when
Adversarial training	Augments training data with adversarial examples so the model learns to resist them	"Usually comes at the cost of decreased model accuracy on clean data" and is expensive to run	A predictive model makes security or fraud decisions and you control training
Randomised smoothing	Adds noise and votes, giving a certified radius for some inputs	Certifies only a subset of test samples	You need a provable bound for a narrow input type
Formal verification	Proves properties of a network over an input region	Limited by scalability and computational cost	Small, safety-critical models
Adversarial input detection	Flags inputs that look crafted (ATLAS AML.M0015 Predictive AI Adversarial Input Detection)	Carlini and Wagner bypassed 10 detection methods; detectors must face adaptive attacks	As a tripwire feeding monitoring, not as the only control
Model hardening and ensembles	Robust architectures and several models that must agree (ATLAS AML.M0003 Predictive AI Model Hardening)	Tramer et al. circumvented 13 existing defences; extra models raise attacker cost rather than prevent evasion	High-value decisions where cost to the attacker matters

NIST's summary is that models which resist evasion while keeping accuracy remain "an open problem". Treat adversarial robustness as a property you measure through red teaming with adaptive attacks. Design the system so one wrong prediction is not decisive: borderline cases go to a human, high-value decisions need a second signal, and rate limits slow query-hungry black box attacks. OWASP ML01:2023 Input Manipulation Attack gives similar advice. See the AI Red Teaming Guide.

Model inversion and membership inference, briefly

A model inversion attack reconstructs sensitive features of training data from a model's outputs; ATLAS AML.T0024.001 Invert AI Model describes strategically querying confidence scores to do so. Membership inference tells an attacker whether a specific record was in the training set. Both get much easier when the API returns confidence scores, which is why the joint deployment guidance says to "ensure APIs return only the minimal data required for the task to inhibit model inversion." Differential privacy, the legal analysis and machine unlearning are covered in the AI and Privacy Guide.

Inference API security

The inference API is the attack surface for extraction, distillation, black box evasion, inversion and plain prediction API abuse. LLM-specific limits on tokens, context and spend are in the LLM Security Guide; the table applies to any model API.

Control	What to set	Attacks it slows
Authenticated callers	No anonymous inference; one API key or OAuth client per customer and application, tied to an account you can suspend	All of them; makes attribution and quotas possible
Model API rate limiting and quotas	Per-caller requests per minute, daily query budgets and spend caps, as OWASP LLM10 and ATLAS AML.M0004 Limit AI Service Query Volume and Rate recommend	Extraction, distillation, decision based evasion, cost harvesting
Confidence score masking	Return the label only, or top-1 with a coarse band (high, medium, low), or scores rounded to one decimal; never full probability vectors by default	Score based evasion, inversion, membership inference, extraction
Restrict logprobs and logit_bias	OWASP: "Restrict or obfuscate the exposure of logit_bias and logprobs in API responses"; enable per client only with a recorded reason	Extraction and side channels on generative models
Strip debug and internal fields	No model version hashes, raw feature vectors or intermediate outputs in responses, errors or client-visible logs (ATLAS AML.T0063)	Reconnaissance and output discovery
Output watermarking	OWASP: "Implement watermarking frameworks to embed and detect unauthorized use of LLM outputs"	Helps prove distillation after the fact; does not prevent it
Usage terms and enforcement	Contract terms that prohibit using outputs to train competing models, backed by monitoring and account suspension	Distillation by identifiable customers

Self-hosted inference servers are part of this surface: the ShadowRay attacks on Ray clusters led to stolen cloud credentials and AI tokens. Put model servers behind an authenticating gateway and keep management ports off the internet. See the API Security Guide.

Choosing controls for your situation

Situation	Main risk	Priority controls
You consume a third-party model API	Leaked API keys and abuse of your account, not theft of the provider's model	Key vaulting and scoping, spend alerts. See the LLMjacking Guide
You self-host open-weight models	Tampered weights, exposed serving infrastructure	Hash and signature checks, authenticated serving
You expose a proprietary model through an API	Extraction, distillation, black box evasion, inversion	All inference API controls above, plus abuse monitoring and usage terms
You train high-value or frontier models	Weight theft by capable or insider attackers	Vaulted weights, HSM keys, minimal human access, insider threat programme, confidential computing
You ship models to devices or customers' premises	Weights are in attackers' hands	Assume white box access; keep the most valuable logic server-side
Predictive model makes security or fraud decisions	Evasion by motivated adversaries	Adversarial training, input detection, second signals, human review thresholds, adaptive red teaming

For securing AI models on a budget, a small organisation exposing one model should start with authentication, quotas and output minimisation, which cost little. Large organisations need a central model inventory first; OWASP recommends "a centralized ML model inventory or registry for models used in production". See the AI Security Posture Management (AI-SPM) Guide.

Implementing AI model security

Inventory models and copies: list every production model, its owner, every place the weights live, who can read them and which APIs expose it.
Lock down weights first: one registry or vault per environment, customer-managed keys, no human read access to production weights, per-workload identities. Test in non-production first; export bucket and key policies beforehand so you can roll back.
Authenticate and meter every inference endpoint: require a caller identity, run quotas in log-only mode for two to four weeks to learn normal volumes, then enforce on a pilot ring of new or free-tier customers before everyone.
Minimise outputs: default to labels or coarse bands and disable logprobs where not needed. Warn consumers, offer an exception path, and keep a feature flag to restore the old format if an integration breaks.
Harden decision models: run adaptive adversarial testing, add adversarial training where the accuracy budget allows, and add human review thresholds. Keep the previous model version deployable as the rollback.
Monitor and respond: send inference and registry logs to the SIEM, build the detections below, and add model theft scenarios, with key rotation and account suspension, to incident response plans.

Detection and measurement

The joint deployment guidance says to "Collect logs to cover inputs, outputs, intermediate states, and errors" and to "Monitor for attempts to access or elicit data from the AI model or aggregate inference responses." ATLAS lists this as AML.M0024 AI Telemetry Logging. Useful detections:

Extraction pattern: one caller with high volume, synthetic-looking inputs, systematic coverage of the input space or inputs clustered near decision boundaries; the guidance flags "high frequency or repetitive inputs".
Distillation pattern: sustained high output volume with broad topic coverage, or many accounts sharing payment details or infrastructure.
Evasion pattern: bursts of near-duplicate inputs with small variations, and rising rates of borderline scores.
Weight access: object storage data events and key usage logs (for example AWS CloudTrail data events and KMS Decrypt calls) on weight paths, alerting on any reader outside the expected workload identities.
Integrity: compare deployed weight hashes with the registry record at start-up.

Metrics that show the programme is working:

Percentage of production models with weights in an approved, encrypted store and a named owner (target 100%).
Number of human identities with read access to production weights (trend down; review quarterly).
Percentage of inference endpoints with caller authentication, per-caller quotas and minimised outputs.
Mean time to detect and suspend an account showing extraction or distillation patterns.
Robust accuracy under adaptive attack for decision models, tracked against clean accuracy at each release.
Percentage of model loads verified against a recorded hash or signature.

Framework mapping

Control area	MITRE ATLAS mitigation	OWASP	NIST SP 800-53 Rev. 5	ISO/IEC 27001:2022 Annex A
Weight access control	AML.M0005 Control Access to AI Models and Data at Rest; AML.M0019 Control Access to AI Models and Data in Production	ML05:2023 Model Theft; LLM10 access controls	AC-3, AC-6	5.15, 8.2, 8.3
Encryption and keys	AML.M0005	ML05:2023 encryption	SC-12, SC-28	8.24
Query limits and output minimisation	AML.M0004 Limit AI Service Query Volume and Rate	LLM10:2025 rate limiting, logits and logprobs	SC-5, AC-4	8.6, 8.26
Evasion resistance	AML.M0003 Predictive AI Model Hardening; AML.M0015 Predictive AI Adversarial Input Detection	ML01:2023 Input Manipulation	SI-10, SA-11	8.29
Logging and monitoring	AML.M0024 AI Telemetry Logging	LLM10 logging and anomaly detection	AU-2, AU-6, SI-4	8.15, 8.16
Model inventory	Not a single mitigation	LLM10 centralised model inventory	CM-8	5.9

In NIST CSF 2.0, inventory sits in ID.AM, weight and API protection in PR.AA and PR.DS, and extraction monitoring in DE.CM. See the MITRE ATLAS Guide.

Common failure patterns

Weights in a shared bucket: production weights sit in a data lake bucket that dozens of analysts and pipelines can read.
Tokens with model repository rights in code: a registry or hub token is committed or baked into an image, as in the Hugging Face token exposure, giving outsiders read or write access to models.
Full probability vectors by default: the API returns every class score or logprob because it was useful in testing, cheapening extraction, inversion and evasion.
Quotas per IP, not per identity: limits keyed on source address are evaded by spreading queries across cloud hosts.
Robustness claimed from one test: a model is called robust after testing against a fixed attack set, which NIST warns has repeatedly been broken by adaptive attacks.
Unauthenticated model servers: inference and orchestration services exposed on the internet without authentication, the pattern behind the ShadowRay campaign.

Practitioner checklist

Inventory production models with owner, weight locations, readers and exposing APIs.
Store weights in one encrypted, access-controlled store per environment with HSM-backed customer-managed keys.
Remove standing human read access to production weights; use per-workload, short-lived identities.
Verify weight hashes or signatures at every load and alert on mismatches.
Require authentication on every inference endpoint and enforce per-identity quotas and spend caps.
Return labels or coarse bands by default; restrict confidence scores, logprobs and logit_bias to approved callers.
Monitor for extraction, distillation and evasion patterns and for unusual weight reads.
Test decision models with adaptive adversarial attacks before each release and track robust accuracy.

AI Model Security Guide: Model Theft, Extraction and Inversion