AI Security Glossary: LLM and Model Attack Terms

AI security has borrowed words from machine learning research, application security and safety work, and the same term often means different things to a data scientist, a penetration tester and a risk committee. This glossary gives short, precise definitions of the terms that matter when securing models and LLM applications: attacks on training data and models, attacks through prompts and retrieved content, privacy attacks, and the controls and frameworks used against them. It is written for security engineers, architects, auditors and risk teams who need to read AI threat reports and vendor claims critically. Where a term comes from a standard or framework, we name it: the OWASP Top 10 for LLM Applications 2025, NIST AI 100-2 E2025 (the adversarial machine learning taxonomy), NIST AI 600-1 (the generative AI profile) and MITRE ATLAS. Terms about AI agents, delegation and agent identity are in the separate Agentic AI Glossary.

A

Adversarial example: An input that has been deliberately altered, often in ways a person would not notice, so that a model produces a wrong or attacker-chosen output. A few changed pixels can make an image classifier mislabel a stop sign; a crafted string can push a text classifier past a filter. MITRE ATLAS records the creation of such inputs as AML.T0043 Craft Adversarial Data. See the Adversarial Examples and Evasion Attacks Guide.
Adversarial machine learning (AML): The study of attacks that exploit how machine learning systems learn and infer, and of the defences against them. NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (March 2025), organises the field by type of ML method, lifecycle stage of the attack, and the attacker's goals, capabilities and knowledge. It is the most useful single reference for agreeing vocabulary across teams. See the Adversarial ML Taxonomy Guide.
AI-BOM (AI bill of materials): A structured inventory of the components behind an AI system: base and fine-tuned models, their versions and sources, datasets, adapters, libraries and the services the system calls. It extends the software bill of materials idea to models and data, so that when a model or dataset is found to be compromised, affected systems can be found quickly. See the AI Supply Chain and AI-BOM Guide.
AI red teaming: Structured adversarial testing of an AI system to find ways it can be made to fail, leak, misbehave or cause harm, covering both security failures (prompt injection, data leakage, tool misuse) and safety failures (harmful content). Good red teaming is scoped to the system as deployed, including its retrieval sources and tools, not only the base model. See the AI Red Teaming Guide.
AI safety and AI security: Related but different disciplines. AI security protects the system, its data and its users from deliberate attack; AI safety is concerned with the system causing harm through its own behaviour, even when nobody is attacking it. The two meet in areas such as jailbreaks, where an attacker defeats safety training. See the AI Safety vs AI Security Guide.
AI supply chain: Everything an AI system depends on from outside the team that runs it: pre-trained models, adapters, datasets, model hubs, ML libraries, hosted model APIs and plug-ins. OWASP LLM03:2025 notes that ML supply chain risk extends beyond code to third-party models and data, which can be tampered with or poisoned. NIST AI 600-1 lists the related risk as Value Chain and Component Integration. MITRE ATLAS records attacks on it as AML.T0010 Supply Chain Compromise. See the AI Supply Chain and AI-BOM Guide.
ATLAS (MITRE ATLAS): MITRE's Adversarial Threat Landscape for Artificial-Intelligence Systems: a knowledge base of tactics, techniques and case studies for attacks on AI systems, modelled on ATT&CK. Technique IDs take the form AML.T0051 (LLM Prompt Injection). Use it to tag detections and threat models in the same way ATT&CK is used for enterprise attacks. See the MITRE ATLAS Guide.

B to D

Backdoor (trojan) attack: An attack that plants hidden behaviour in a model so that it acts normally on ordinary inputs but produces an attacker-chosen output when a specific trigger appears, such as a phrase, a pixel pattern or a code comment. Backdoors are usually inserted through poisoned training or fine-tuning data or by tampering with model weights, and they survive standard accuracy testing because the model is accurate until triggered. MITRE ATLAS covers model tampering under AML.T0018.000 Poison AI Model. See the AI Data Poisoning and Backdoors Guide.
Confabulation: NIST AI 600-1's term for what is commonly called hallucination: the production of confidently stated but erroneous or false content. NIST prefers it because the model is not perceiving anything; it is generating plausible text. Confabulation is one of the 12 risks the generative AI profile lists. See the AI Hallucination Business Risk Guide.
Content credentials (C2PA manifest): A C2PA manifest, also called a Content Credential, is a set of provenance information about a digital asset made of one or more assertions that are digitally signed so their authenticity and integrity can be checked. Content credentials can show that an image or video was produced or edited with an AI tool. They prove what a signer claimed, not that the content is true, and they can be stripped from a file. See the AI Content Provenance and C2PA Guide.
Data poisoning: Manipulation of pre-training, fine-tuning or embedding data to introduce vulnerabilities, backdoors or biases into a model (OWASP LLM04:2025 Data and Model Poisoning). It is an integrity attack, and it can happen at any stage that ingests data, including scraping from the public web and user feedback loops. MITRE ATLAS records it as AML.T0020 Training Data Poisoning. See the OWASP LLM Top 10 LLM04 Data and Model Poisoning Guide.
Deepfake: Synthetic audio, video or images generated or altered by AI to impersonate a real person or fabricate an event. In security, the main concern is fraud and social engineering: cloned executive voices approving payments, and synthetic faces used to pass identity verification. Controls sit in process (call-back verification, payment approvals) as much as in detection. See the Deepfake and AI Impersonation Guide.
Denial of wallet: An attack, or an accident, that drives up the cost of a pay-per-use AI service by forcing large numbers of expensive inference calls or very long inputs and outputs, so the impact is financial rather than an outage. It is one of the outcomes OWASP groups under LLM10:2025 Unbounded Consumption, and MITRE ATLAS records the cost-focused variant as AML.T0034 Cost Harvesting. See the Denial-of-Wallet and AI Agent Cost Runaway Guide.
Differential privacy: A mathematical guarantee that the output of an analysis, or a model trained on data, changes only by a bounded amount whether or not any one individual's record is included. It is achieved by adding calibrated noise during training or querying, controlled by a privacy budget usually written as epsilon. In AI it is used to limit membership inference and training data extraction, at some cost to accuracy. See the Differential Privacy Guide.
Direct prompt injection: Prompt injection in which the attacker types the malicious instructions straight into the model's input, for example in a chat window or an API call, to override the system prompt or make the application behave in unintended ways. It is the form most people meet first. MITRE ATT&CK v20 lists it as T1699 Direct Prompt Injection and ATLAS as AML.T0051.000. See the MITRE ATT&CK T1699 Direct Prompt Injection guide.

E to H

Embedding: A list of numbers (a vector) that represents the meaning of a piece of text, an image or other data, produced by an embedding model so that similar content sits close together in vector space. Retrieval systems compare embeddings to find relevant documents. Embeddings are derived from the source data and can leak information about it, so they need the same access controls as the documents they represent. See the OWASP LLM Top 10 LLM08 Vector and Embedding Weaknesses Guide.
Evasion attack: An attack at inference time that crafts inputs so a deployed model misclassifies them, for example malware altered so an ML-based detector scores it as benign, or content altered to slip past a moderation classifier. Unlike poisoning, evasion does not change the model; it exploits the model's decision boundary. MITRE ATLAS records it as AML.T0015 Evade AI Model. See the Adversarial Examples and Evasion Attacks Guide.
Excessive agency: OWASP LLM06:2025: the vulnerability that lets damaging actions be performed in response to unexpected, ambiguous or manipulated outputs from an LLM, whatever is causing the model to malfunction. OWASP names three root causes: excessive functionality, excessive permissions and excessive autonomy. It is where prompt injection turns into real-world impact. See the OWASP LLM Top 10 LLM06 Excessive Agency Guide.
Federated learning: A training approach in which a model is trained across many devices or organisations that keep their data locally and share only model updates. It reduces central data collection, but the updates themselves can leak information and can be poisoned by a malicious participant, so it is often combined with secure aggregation and differential privacy. See the Federated Learning Privacy guide.
Fine-tuning: Further training of a pre-trained model on a smaller, task-specific dataset to change its behaviour or knowledge. Parameter-efficient methods such as LoRA (Low-Rank Adaptation) train small adapter weights instead of the whole model. OWASP LLM03:2025 notes that these methods, and the ease of sharing adapters on public model hubs, add supply chain risk. Fine-tuning data is also a poisoning target and can carry sensitive records into the model. See the AI Model Security Guide.
Guardrails: Controls placed around a model to filter, block or rewrite inputs and outputs: prompt injection classifiers, sensitive data detectors, topic restrictions and output validators. They reduce risk but are probabilistic and can be bypassed, so they complement deterministic controls such as least privilege rather than replace them. See the AI Guardrails and AI Firewalls Guide.
Hallucination: Content generated by a model that seems accurate but is fabricated. OWASP LLM09:2025 Misinformation names hallucination as a major cause of false or misleading output and pairs it with overreliance, where people or systems act on output without checking it. In security, a hallucinated package name or API can become an attack path when attackers register the name. NIST uses the term confabulation for the same behaviour. See the OWASP LLM Top 10 LLM09 Misinformation Guide.

I to L

Improper output handling: OWASP LLM05:2025: insufficient validation, sanitisation and handling of LLM output before it is passed to other components. Because output can be steered through the prompt, passing it unchecked into a browser, shell, database query or API gives the attacker indirect access to those systems, with outcomes such as cross-site scripting, server-side request forgery or remote code execution. Treat model output as untrusted user input. See the OWASP LLM Top 10 LLM05 Improper Output Handling Guide.
Indirect prompt injection: Prompt injection carried in content the model reads rather than in what the user types: a web page, email, document, calendar invite, code comment or tool result. The user may never see the instruction. It is the more dangerous form for assistants and agents that read external content and hold permissions. MITRE ATT&CK v20 lists it as T1698 Indirect Prompt Injection and ATLAS as AML.T0051.001. See the MITRE ATT&CK T1698 Indirect Prompt Injection guide.
Inference: The use of a trained model to produce an output from an input, as opposed to training. Most AI attacks against deployed systems happen at inference time through the inference API: evasion, prompt injection, extraction and membership inference. Rate limits, authentication and logging on the inference endpoint are therefore core security controls. See the LLM Security Guide.
Jailbreak: A form of prompt injection in which the attacker gets the model to disregard its safety training or usage rules, for example through role-play, hypothetical framing, encoded text or many-turn escalation. OWASP LLM01:2025 treats jailbreaking as a subset of prompt injection; MITRE ATLAS lists it separately as AML.T0054 LLM Jailbreak. A jailbreak defeats the model's own restrictions, which is why access to data and tools must be limited outside the model. See the LLM Jailbreaks Guide.
Large language model (LLM): A neural network trained on very large amounts of text to predict the next token, which lets it generate and transform language, code and structured data. For security, the defining property is that an LLM processes instructions and data in the same input stream, which is the root of prompt injection. See the LLM Security Guide.
LLMjacking: The theft and abuse of credentials for paid AI services, typically cloud API keys or access keys, so attackers can run inference at the victim's expense or resell access. It is a non-human identity problem first: the defence is keeping AI service keys out of code, scoping and rotating them, and alerting on unusual model usage and spend. See the Denial-of-Wallet and AI Agent Cost Runaway Guide.

M

Membership inference: A privacy attack that determines whether a specific record was in a model's training data, usually by comparing how confidently the model responds to that record against others. If the training set is sensitive, such as patients with a condition, membership itself is the disclosure. MITRE ATLAS records it as AML.T0024.000 Infer Training Data Membership. See the Membership Inference Attacks guide.
Model card: A short document published with a model describing its intended use, training data at a summary level, evaluation results, known limitations and ethical considerations. For security and procurement teams, the model card is the first evidence of what a third-party model was built for and tested against, and a missing card is a due diligence finding. See the AI Model Provenance and Watermarking Guide.
Model extraction (model stealing): Recreating a functionally similar copy of a model by querying it at scale and training a substitute on its answers, sometimes called distillation when done deliberately. It steals intellectual property and gives the attacker an offline copy on which to develop evasion or jailbreak attacks. MITRE ATLAS records it as AML.T0024.002 Extract AI Model; OWASP lists it among the impacts of LLM10:2025 Unbounded Consumption. See the Model Extraction and Distillation Defence Guide.
Model inversion: A privacy attack that reconstructs features of the training data from a model's outputs, for example recovering a recognisable face from a facial recognition model's confidence scores. It differs from membership inference, which only answers whether a record was present. MITRE ATLAS records it as AML.T0024.001 Invert AI Model. See the Adversarial ML Taxonomy Guide.
Model poisoning: Direct tampering with a model's weights or architecture, rather than its training data, to change its behaviour, often to implant a backdoor. It is most likely where models are downloaded from public hubs or passed between teams without integrity checks. OWASP covers it with data poisoning in LLM04:2025. See the AI Model Poisoning Incident Playbook.
Model serialisation attack: Hiding executable code in a model file so it runs when the file is loaded. Many model formats rely on Python's pickle, and the Python documentation warns that pickle is not secure: malicious pickle data can execute arbitrary code during unpickling. Loading an untrusted model can therefore compromise the host and any credentials on it. MITRE ATLAS covers it under AML.T0011.000 Unsafe AI Artifacts. Prefer formats that store only tensors, such as safetensors, and scan models before use. See the Hugging Face Security Guide.
Model weights: The learned numerical parameters that define a trained model. Whoever holds the weights holds the model: they can run it, fine-tune it and probe it offline without rate limits. Weights of proprietary or fine-tuned models are high-value assets that need access control, encryption and monitoring like any crown-jewel data. See the Securing AI Model Weights guide.
Multimodal prompt injection: Prompt injection hidden in images, audio or video that a multimodal model processes, such as text rendered faintly in an image or instructions embedded in a document scan. OWASP LLM01:2025 notes that injected content does not need to be readable by a human, only parsed by the model. Text-only filters do not see it. See the Multimodal Prompt Injection Guide.

N to P

Open-weight model: A model whose trained weights are published for download, so anyone can run, inspect and fine-tune it. Open weights allow private deployment without sending data to a provider, but they also let attackers study the model offline and remove safety fine-tuning, and they make the download path part of the supply chain. Open weights are not the same as open source: training data and code may not be released. See the OWASP LLM Top 10 LLM03 Supply Chain Guide.
Overreliance: Trusting LLM output without adequate checking, so that errors flow into decisions, code or customer communications. OWASP discusses it alongside hallucination under LLM09:2025 Misinformation. The control is human review proportionate to impact and verification steps in the workflow, not better prompts alone. See the AI Hallucination Business Risk Guide.
Prompt obfuscation: Disguising malicious instructions so that filters and reviewers miss them while the model still follows them: encoding, splitting text across messages, unusual languages, invisible Unicode characters or formatting tricks. It is why deny-lists of bad phrases fail. MITRE ATLAS records it as AML.T0068 LLM Prompt Obfuscation. See the Prompt Injection Guide.
Prompt injection: OWASP LLM01:2025: a vulnerability that occurs when prompts alter the LLM's behaviour or output in unintended ways, including content that is imperceptible to humans. It is the top risk because an LLM has no reliable boundary between instructions and data. The UK NCSC has warned that, unlike SQL injection, prompt injection may never be fully mitigated, and recommends designing so that a manipulated model cannot do serious harm. MITRE ATLAS records it as AML.T0051 LLM Prompt Injection. See the Prompt Injection Guide.

R to S

Retrieval-augmented generation (RAG): A pattern in which the application retrieves relevant documents, usually by embedding similarity from a vector database, and adds them to the prompt so the model answers from that content. RAG brings two security problems: retrieved content can carry indirect prompt injection, and retrieval can return documents the user is not authorised to see unless permissions are enforced at retrieval time. See the RAG Security Guide.
RAG poisoning: Planting crafted content in a knowledge source that a RAG system indexes, so that it is retrieved for chosen queries and steers the answer or carries injected instructions. Any source that outsiders or many insiders can write to, such as wikis, ticketing systems, shared drives and web forms, is a candidate. MITRE ATLAS records it as AML.T0070 RAG Poisoning. See the RAG Security Guide.
Safetensors: A file format for storing model tensors that its maintainers describe as a simple format for storing tensors safely, as opposed to pickle, while remaining fast through zero-copy loading. Because it holds tensor data and metadata rather than serialised Python objects, loading it is not a route to code execution. Requiring safetensors for third-party models is a simple supply chain control. See the AI Model Scanning and Supply Chain guide.
Sensitive information disclosure: OWASP LLM02:2025: exposure of personal data, financial details, health records, confidential business data, credentials or proprietary model details through an LLM's output. Sources include training data the model memorised, documents retrieved without permission checks and data users paste into prompts. See the OWASP LLM Top 10 LLM02 Sensitive Information Disclosure Guide.
Shadow AI: AI tools, features and models used inside an organisation without approval or visibility from IT, security or governance: personal accounts on public chatbots, browser extensions, AI features switched on in existing SaaS, and unsanctioned API keys. The main risks are data leaving the organisation and decisions made with unassessed tools. See the AI Usage Control and Shadow AI guide.
System prompt: The instructions an application gives a model before any user input, setting its role, rules and output format. The system prompt steers behaviour but does not enforce it: a user or injected content can often override it. Permissions, data access and business rules must be enforced outside the model. See the OWASP LLM Top 10 LLM07 System Prompt Leakage Guide.
System prompt leakage: OWASP LLM07:2025: the risk that system prompts contain sensitive information, such as credentials, connection strings or descriptions of internal roles and permissions, that an attacker can extract. OWASP is explicit that the system prompt should not be treated as a secret or used as a security control. MITRE ATLAS records extraction as AML.T0056 Extract LLM System Prompt. See the OWASP LLM Top 10 LLM07 System Prompt Leakage Guide.

T to Z

TEVV (test, evaluation, verification and validation): The NIST AI RMF term for the activities that check whether an AI system works as intended and meets its trustworthiness goals, across its lifecycle rather than only before release. Security testing and red teaming are part of TEVV, alongside performance, robustness and bias evaluation. See the NIST AI RMF Guide.
Token (LLM): The unit of text a language model reads and writes, typically a word fragment. Context window size, latency and cost are all measured in tokens, which is why very long inputs and outputs are the lever for denial of wallet. Not to be confused with access tokens and API tokens, which are credentials. See the OWASP LLM Top 10 LLM10 Unbounded Consumption Guide.
Training data extraction: Getting a model to reproduce verbatim fragments of its training data, such as personal records, source code or secrets it memorised. It is one route to sensitive information disclosure under OWASP LLM02:2025, and the reason credentials and personal data should be removed from training and fine-tuning sets before use. See the OWASP LLM Top 10 LLM02 Sensitive Information Disclosure Guide.
Unbounded consumption: OWASP LLM10:2025: an LLM application allowing excessive and uncontrolled inference, leading to denial of service, economic loss, model theft or degraded service. Controls include input size limits, rate limits and quotas per identity, spend alerts and timeouts on long-running generations. MITRE ATLAS records the availability impact as AML.T0029 Denial of AI Service. See the OWASP LLM Top 10 LLM10 Unbounded Consumption Guide.
Vector database: A database that stores embeddings and returns the nearest matches to a query embedding, used as the retrieval layer in RAG. OWASP LLM08:2025 Vector and Embedding Weaknesses covers the risks: unauthorised access and data leakage across tenants or permission groups, poisoned entries and embedding inversion. Apply document-level permissions in the vector store, not only in the source system. See the OWASP LLM Top 10 LLM08 Vector and Embedding Weaknesses Guide.
Watermarking (AI output): Embedding a signal in AI-generated text, images or audio so it can later be identified as machine-generated, either imperceptibly in the content itself or through attached metadata. Watermarks can be weakened by editing, compression or paraphrasing, so they are evidence rather than proof, and they work best alongside signed provenance such as C2PA. See the AI Model Provenance and Watermarking Guide.
Zero-click prompt injection: Indirect prompt injection that needs no action from the victim beyond the assistant doing its normal work: an email, invite or record arrives, the AI assistant reads it while summarising or answering, and the hidden instructions run. It shows why assistants that read untrusted inbound content should not also hold broad data access and outbound channels at the same time. The EchoLeak 2025 case is a documented example. See the Prompt Injection Incident Response Playbook.

For agent, delegation and agent identity terms, see the Agentic AI Glossary, and for identity and non-human identity terms the NHI and IAM Glossary. Related NHI Mgmt Group resources: the AI Threats and Model Security Guide, the LLM Security Guide and the MITRE ATLAS Guide.

AI Security Glossary: Plain Definitions of LLM and Model Attack Terms

A