AI systems can be attacked through their training data, their model files, their inference interfaces and the prompts and content they read, and most of these attacks leave no trace in conventional security tooling. A poisoned dataset, a backdoored fine-tune, an extracted model or an indirect prompt injection does not look like malware on an endpoint, yet each can change what an AI system does or leak what it knows. This page is the overview of the AI Threats and Model Security category: the attack classes defined by NIST AI 100-2 E2025, OWASP and MITRE ATLAS, the controls that work at each maturity level, who owns them, and the frameworks to map against. It summarises the field and links every NHI Mgmt Group guide in the category for depth. It is written for AI security engineers, ML platform owners, red teams, threat intelligence analysts and the security architects who approve AI systems. Attacks are described only at the level needed to defend against them.
Key takeaways
- Four attacker objectives cover the field. NIST AI 100-2 E2025 groups attacks by goal: availability breakdown, integrity violation, privacy compromise and misuse, across both predictive and generative AI.
- There is no complete technical fix. NIST describes most AI mitigations as empirical and notes that resisting evasion while keeping accuracy remains an open problem, so defence in depth and limits on what a model can reach matter most.
- Model files and datasets are supply chain. Pickle-based model formats can run code when loaded, and poisoned data or weights arrive through hubs, registries and packages, so provenance and signing beat after-the-fact detection.
- Inference APIs are an attack surface. Extraction, membership inference, jailbreaks and cost harvesting all work through the normal API, so authentication, rate limits and query monitoring are security controls.
- Credentials sit behind most real incidents. Stolen API keys, CI tokens and machine credentials appear in LLMjacking, the LiteLLM package compromise and the 2026 OpenAI and Hugging Face evaluation incident.
What the field covers
This category is about attacks on AI systems themselves and the security of models across their lifecycle: data collection, training, fine-tuning, evaluation, packaging, deployment and inference. It also covers attackers using AI to attack everything else. It sits between two neighbours: the AI Platforms category (securing specific products such as Bedrock or Copilot) and the agent categories (autonomous systems that take actions). Prompt injection appears in all three because it is the threat that connects them.
| Area | Question it answers | Example |
|---|---|---|
| Training-time integrity | Can someone change what the model learns? | Data poisoning, backdoors, malicious fine-tunes |
| Model artefact security | Is the file we load safe and the one we approved? | Pickle code execution, tampered weights, unsigned adapters |
| Inference-time attacks | Can inputs make the model misbehave? | Evasion examples, jailbreaks, direct and indirect prompt injection |
| Model and data privacy | Can someone recover the model or its training data? | Model extraction, membership inference, data reconstruction |
| Availability and cost | Can someone exhaust or misuse the service? | Denial of AI service, cost harvesting with stolen keys |
| Offensive use of AI | How do attackers use AI against us? | AI-assisted phishing, deepfakes, faster exploit development |
Core concepts
- Predictive versus generative AI: NIST AI 100-2 E2025 treats them separately. Predictive AI attacks centre on evasion, poisoning and privacy; generative AI adds supply chain, direct prompting, indirect prompt injection and misuse.
- Attacker knowledge: white-box attackers have the weights (as with open-weight models, ATLAS AML.T0044 Full AI Model Access); black-box attackers only have the inference API (AML.T0040 AI Model Inference API Access). Open weights move many attacks from hard to easy.
- Lifecycle stage: poisoning acts at training or fine-tuning time; evasion, injection and extraction act at deployment time. Controls must be placed at the stage where the attack enters.
- Data poisoning versus model poisoning: the first controls part of the training data; the second modifies the model or its updates directly, for example a tampered checkpoint or adapter.
- Backdoors: hidden behaviour that appears only with a trigger. Research summarised in the AI Data Poisoning and Backdoors Guide found that a small number of poisoned documents can be enough and that backdoors can survive safety training.
- Trust boundary in the context window: an LLM cannot reliably tell instructions from data, so any untrusted content placed in its context (email, web pages, retrieved documents, tool output) can act as instructions.
Main threats
| Threat | NIST AI 100-2 class | MITRE ATLAS | OWASP LLM 2025 | Primary defence |
|---|---|---|---|---|
| Training data poisoning and backdoors | Poisoning (availability, targeted, backdoor) | AML.T0020 Poison Training Data | LLM04 Data and Model Poisoning | Dataset provenance, curation, held-out evaluation |
| Model tampering | Model poisoning | AML.T0018 Manipulate AI Model | LLM04 | Signed artefacts, hash pinning, controlled registries |
| Malicious or compromised model supply chain | GenAI supply chain | AML.T0010 AI Supply Chain Compromise, AML.T0011 User Execution | LLM03 Supply Chain | Safe formats such as safetensors, scanning, AI-BOM |
| Evasion (adversarial examples) | Evasion | AML.T0043 Craft Adversarial Data | Not a separate entry | Adversarial testing, input validation, ensembles, human review for high-stakes decisions |
| Direct prompt injection and jailbreaks | Direct prompting | AML.T0051.000 Direct, AML.T0054 LLM Jailbreak | LLM01 Prompt Injection | Layered guardrails, least privilege on tools and data |
| Indirect prompt injection | Indirect prompt injection | AML.T0051.001 Indirect | LLM01 | Treat retrieved content as untrusted, confirmation for actions, output egress control |
| Model extraction | Privacy: model extraction | AML.T0024.002 Extract AI Model | LLM10 Unbounded Consumption | Authentication, rate limits, query pattern monitoring |
| Membership inference and data reconstruction | Privacy | AML.T0024.000 Infer Training Data Membership, AML.T0057 LLM Data Leakage | LLM02 Sensitive Information Disclosure | Data minimisation, deduplication, differential privacy where feasible |
| System prompt extraction | Privacy (GenAI) | AML.T0056 Extract LLM System Prompt | LLM07 System Prompt Leakage | No secrets in prompts, enforce rules outside the model |
| Denial of AI service and cost harvesting | Availability | AML.T0029 Denial of AI Service, AML.T0034 Cost Harvesting | LLM10 | Quotas, budgets, key hygiene |
Real incidents show how often the AI-specific technique is paired with an ordinary credential failure. In LLMjacking, attackers used stolen cloud credentials and API keys to run models on victims' accounts (ATLAS AML.T0055 Unsecured Credentials). The LiteLLM PyPI breach was an AI supply chain compromise: a poisoned security scanner in CI exposed a publishing token, and the malicious releases harvested LLM provider keys and cloud credentials. In the OpenAI and Hugging Face breach 2026, agents in a cyber evaluation escaped their sandbox and, according to both companies, harvested Kubernetes tokens, cloud role credentials and a JWT signing key. For the attacker's use of AI, see the AI-Powered Cyber Attacks Guide.
Choosing where to focus
Exposure depends on how you use AI. Prioritise by the deployment pattern rather than by the full threat catalogue.
| How you use AI | Threats that dominate | Start with |
|---|---|---|
| Consume hosted models through APIs or SaaS | Prompt injection, data leakage, stolen keys | Key management, data scoping, output handling, logging |
| Build RAG applications | Indirect injection, retrieval of over-permissioned data, embedding weaknesses | Permission-aware retrieval and untrusted-content handling; see the RAG Security Guide |
| Fine-tune hosted or open models | Poisoned tuning data, malicious adapters, safety regression | Data provenance, adapter signing, regression evaluation; see the Fine-Tuning and LoRA Security Guide |
| Run open-weight models | Malicious model files, white-box attacks, removed safety tuning | Safe formats, scanning, isolated loading; see the Open-Weight Model Security Guide |
| Train or serve proprietary models | Extraction, membership inference, poisoning, weight theft | Weight access control, API rate limits, privacy testing; see the AI Model Security Guide |
Smaller organisations mostly consume hosted models; their priorities are keys, data scoping and supplier assurance, with red teaming bought in for high-risk applications. Larger organisations that train or fine-tune need an ML supply chain programme (dataset registry, model registry with signing, AI-BOM), an internal AI red team and detections on inference APIs.
Key controls by maturity
| Area | Foundational | Managed | Advanced |
|---|---|---|---|
| Inventory and provenance | List of models, datasets and sources in use | Model and dataset registry with owner, version and hash | AI-BOM per system with signed artefacts; see the AI Supply Chain and AI-BOM Guide |
| Artefact safety | Block pickle-based formats from untrusted sources | Scan model files before load; load in isolated environments | Signature verification (ATLAS AML.M0013 Code Signing) enforced in deployment pipelines |
| Access to models and data | Authentication on every inference endpoint | Least privilege on weights, datasets and training jobs (AML.M0005) | Short-lived workload credentials and separate identities per pipeline stage |
| Input and output controls | Provider guardrails on; no secrets in system prompts | Prompt attack detection (AML.M0015 Adversarial Input Detection), output sanitisation, egress control on rendered links | Context isolation patterns that separate untrusted content from tool-using steps |
| Testing | Pre-release safety and security tests on critical use cases | Scheduled AI red teaming mapped to ATLAS and OWASP | Continuous automated evaluation and regression on every model or prompt change |
| Monitoring | Usage and spend alerts | Inference logging, abuse and extraction pattern detection | AI incident playbooks exercised with the SOC |
Implementation approach
- Threat model each AI system: map its data sources, model origin, interfaces and tools, then walk the ATLAS tactics to find where an attacker could enter. See the MITRE ATLAS Guide.
- Secure the supply chain first: pin and hash every model, dataset and AI library, prefer safetensors, and require approval for new sources.
- Lock down credentials: move provider keys and training-job credentials into a secrets manager or workload identity, scope CI secrets per job, and rotate anything a compromised host could reach.
- Test in a sandbox: run red team and evaluation suites in isolated environments with no route to production credentials or the internet, a lesson underlined by the 2026 evaluation incidents. See the AI Red Teaming Guide.
- Deploy behind controls: authentication, quotas, guardrails, output handling and logging enabled before go-live; keep the previous model version deployable.
- Roll back on regression: if evaluation or monitoring shows a backdoor trigger, safety regression or abuse, revert to the last signed model and dataset version, revoke exposed keys and preserve logs and artefacts for investigation.
Roles and ownership
| Role | Owns |
|---|---|
| ML platform owner | Model and dataset registries, training infrastructure, artefact signing and rollback |
| AI security engineering | Threat models, guardrail design, output handling, security requirements for AI releases |
| AI red team | Adversarial testing against ATLAS and OWASP, jailbreak and injection testing, findings tracking |
| Data owners and privacy | Training data provenance, consent and lawful basis, privacy testing for memorisation; see the Synthetic Data Risk Guide for synthetic data |
| Identity and secrets teams | API keys, CI tokens, service accounts and workload identities used by pipelines and inference |
| SOC and threat intelligence | Inference and spend detections, AI threat intelligence, AI incident response |
| AI governance or risk | Risk acceptance, release approval for high-risk models, reporting; see the AI Governance Guide |
Where identity fits
Model security is usually discussed as a data science problem, but the paths attackers take into AI systems are mostly identity paths. Training jobs run under service accounts that can read datasets and write checkpoints; CI pipelines hold tokens that publish packages and models; inference applications call providers with API keys; model hubs issue access tokens for private repositories; and agents under evaluation can inherit whatever credentials their runtime exposes. ATLAS reflects this with techniques such as AML.T0055 Unsecured Credentials, and ATT&CK techniques such as T1078.004 Cloud Accounts and T1552.001 Credentials In Files appear in the same incident chains.
- Write access to datasets and registries is the poisoning control: whoever can change a training set, a model registry entry or an adapter can plant a backdoor, so these permissions deserve privileged-access treatment, with approval and logging.
- Separate identities per pipeline stage: ingestion, training, evaluation and deployment should not share one credential, so a compromised step cannot rewrite the model it hands on.
- Short-lived over static: workload identity federation for cloud training and inference removes the long-lived keys that LLMjacking depends on.
- Sandbox credentials: evaluation and red team environments get their own throwaway identities with no trust relationship to production.
See the Agent Identity and Access Guide and the LLMjacking Guide.
Detection and measurement
- What to log: model and dataset downloads and registry changes, training job launches and their credentials, inference requests with caller identity, guardrail and prompt attack detections, tool calls made by models, and spend per key.
- What to alert on: high-volume or systematic query patterns consistent with extraction, unsigned or unscanned artefacts reaching deployment, new model sources, keys used from unexpected networks, sudden spend increases, and links or images in outputs pointing to unknown domains.
| Metric | What good looks like |
|---|---|
| Production models with verified provenance and signature | 100% |
| AI systems with a current threat model mapped to ATLAS | All high-risk systems, reviewed at each major change |
| Red team findings closed within agreed time | Above 90% for high-severity findings |
| Jailbreak and injection success rate in regression tests | Falling release on release, never rising after a change |
| Inference endpoints without authentication | Zero |
| Time to revoke and rotate an exposed AI credential | Hours |
Common failure patterns
- Loading models like data: treating a downloaded checkpoint as inert when pickle-based formats execute code on load.
- Trusting the guardrail: relying on a classifier to stop prompt injection while giving the model broad data and tool access, the gap shown by EchoLeak.
- Shared CI secrets: letting scanners and build steps read publishing tokens and provider keys, as in the LiteLLM compromise.
- Evaluations with production reach: running offensive or agentic evaluations where the model can reach real credentials, registries or the internet.
- Testing once: red teaming before launch, then changing the model, prompt or retrieval sources without regression tests.
- Unauthenticated self-hosted endpoints: inference servers reachable without authentication, inviting cost harvesting and extraction.
Standards and frameworks
| Framework | Role in this field |
|---|---|
| NIST AI 100-2 E2025 (March 2025) | Taxonomy and terminology of adversarial machine learning attacks and mitigations, by AI type, lifecycle stage and attacker goal |
| MITRE ATLAS | Tactics, techniques (AML.T IDs), mitigations and case studies for AI attacks; complements ATT&CK |
| OWASP Top 10 for LLM Applications 2025 | Application-level risks LLM01 to LLM10; see the OWASP Top 10 for LLM Applications Guide |
| NIST AI RMF 1.0 and AI 600-1 | Risk management functions Govern, Map, Measure, Manage, and generative AI risks; see the NIST AI RMF Guide |
| NCSC and partners: Guidelines for secure AI system development | Secure design, development (including supply chain), deployment and operation |
| NIST SP 800-53 Rev. 5 | SR-3 Supply Chain Controls and Processes, SR-11 Component Authenticity, SA-11 Developer Testing and Evaluation, CA-8 Penetration Testing, AC-6 Least Privilege, SI-4 System Monitoring |
| NIST CSF 2.0 | GV.SC-07 supplier risk, ID.RA-01 vulnerabilities identified, PR.AA-05 least privilege, DE.CM-09 monitoring of runtime environments |
| ISO/IEC 42001:2023 | AI management system within which model security controls are governed |
Where to go next (all guides in this category)
Threat frameworks and testing
- MITRE ATLAS Guide: the ATLAS matrix, key techniques, case studies and how to use it for threat modelling and detection.
- AI Red Teaming Guide: scoping, methods and safe execution of adversarial testing for AI systems.
Model and data integrity
- AI Data Poisoning and Backdoors Guide: how poisoning enters the lifecycle, sleeper agent backdoors and provenance controls.
- AI Model Security Guide: model theft, extraction and inversion, and how to protect weights and inference APIs.
- Fine-Tuning and LoRA Security Guide: securing custom tuning data, adapters and the safety of tuned models.
- Open-Weight Model Security Guide: risks and controls for Llama, Mistral, DeepSeek, Qwen and other downloadable models.
- Synthetic Data Risk Guide: privacy leakage, quality problems and model collapse from synthetic training data.
Attackers using AI and the evidence base
- AI-Powered Cyber Attacks Guide: what AI has changed for phishing, deepfakes, malware and exploitation, and how to defend.
- AI Security Statistics Guide: sourced figures on AI adoption, AI-enabled attacks and governance gaps.
Related pillars: the AI Platforms Guide for product-level controls, the LLM Security Guide and Prompt Injection Guide for application threats, and the Agent Threats and Attacks Guide for agentic systems.
Practitioner checklist
- Maintain a registry of every model, adapter and dataset in use, with source, owner, version and hash.
- Block or convert pickle-based model files from untrusted sources and scan all artefacts before loading.
- Verify signatures or hashes for models and datasets in the deployment pipeline, not only at download.
- Threat model each high-risk AI system against MITRE ATLAS and the OWASP LLM Top 10 before release.
- Require authentication, quotas and logging on every inference endpoint, including internal and self-hosted ones.
- Keep secrets out of system prompts and enforce authorisation outside the model.
- Scope CI and training-job credentials per job and replace long-lived AI keys with short-lived credentials.
- Run AI red teaming and evaluations in isolated sandboxes with no path to production credentials.
- Add jailbreak, injection and backdoor regression tests to every model, prompt or retrieval change.
- Keep the last known-good model and dataset deployable, and rehearse rollback with the SOC.
Standards and references
- NIST: AI 100-2 E2025 Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
- MITRE ATLAS: Poison Training Data
- OWASP: Top 10 for LLM Applications 2025
- NIST: AI Risk Management Framework
- NCSC: Guidelines for secure AI system development
- NIST: SP 800-53 Rev. 5
- ISO: ISO/IEC 42001:2023 Artificial intelligence management system
- ENISA: Multilayer Framework for Good Cybersecurity Practices for AI
Related NHI Mgmt Group resources: AI Platforms Guide · MITRE ATLAS Guide · AI Supply Chain and AI-BOM Guide · Prompt Injection Guide · LLMjacking Guide