Risks that arise from how large language models process prompts, context, tools, and data. These threats include prompt injection, sensitive data leakage, unsafe tool invocation, and unintended actions caused by model behaviour or chained integrations. They require testing and controls that go beyond standard API security checks.
Expanded Definition
LLM-specific threats are failure modes that emerge from the way a NIST AI 600-1 Generative AI Profile describes generative systems interacting with prompts, context, tools, and external data. They are not limited to malicious user input. They also include compromised retrieval sources, unsafe tool chaining, output manipulation, and leakage caused by model memory or hidden instructions. In practice, these threats sit at the intersection of model behavior, application design, and identity trust boundaries, which is why standard API testing alone does not cover them.
Definitions vary across vendors when systems blend chat interfaces, retrieval, and automation, but the core idea is consistent: the model can be induced to do something the developer did not intend. That can happen through direct prompt injection, indirect prompt injection in content fetched from third parties, or tool misuse when an agent is allowed to take actions with insufficient guardrails. The most common misapplication is treating LLM-specific threats as generic application bugs, which occurs when teams test only the endpoint and ignore how prompts, tools, and upstream content are combined.
Examples and Use Cases
Implementing LLM-specific threat controls rigorously often introduces latency, workflow friction, and higher review overhead, requiring organisations to weigh automation speed against safety and trust boundaries.
- Prompt injection in a support assistant causes the model to ignore policy and reveal internal instructions, a pattern increasingly discussed in the OWASP Agentic AI Top 10.
- Indirect prompt injection appears in a document or webpage that an LLM retrieves, then the model treats attacker-controlled text as higher priority than the system prompt.
- Unsafe tool invocation occurs when an agent has permission to send emails, create tickets, or modify records without step-up validation for high-impact actions.
- Sensitive data leakage happens when a model repeats secrets, tokens, or personal data from context, memory, or logs into an output channel.
- Threat modeling for these cases often maps well to the CSA MAESTRO agentic AI threat modeling framework and the MITRE ATLAS adversarial AI threat matrix, especially where the system is exposed to adversarial content or chained automation.
Why It Matters for Security Teams
LLM-specific threats matter because they can turn a trustworthy interface into an execution layer for unauthorized action. Security teams need to understand that the risk is not only model hallucination. It is also boundary failure, where the model is allowed to interpret untrusted text, call tools, or move data across trust zones without enough policy enforcement. That makes identity, privilege, and data handling central to the control design, especially when AI agents inherit access from human users or NHI service identities.
For governance, the important shift is from content moderation to operational containment. Controls should align to NIST AI Risk Management Framework principles and the threat patterns surfaced in Anthropic’s first AI-orchestrated cyber espionage campaign report, where AI systems were used to scale malicious activity. Organisations typically encounter the full severity of LLM-specific threats only after a model leaks data, sends an unauthorised action, or becomes an entry point into a broader compromise, at which point the term becomes operationally unavoidable to address.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, CSA MAESTRO and MITRE ATLAS address the attack and risk surface, while NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI RMF frames governance, mapping, measurement, and management for generative AI risks. | |
| NIST AI 600-1 | The Generative AI Profile identifies risks from prompts, context, tools, and outputs. | |
| OWASP Agentic AI Top 10 | OWASP documents agentic AI risks such as prompt injection and unsafe autonomy. | |
| CSA MAESTRO | MAESTRO provides agentic AI threat modeling for autonomous and tool-using systems. | |
| MITRE ATLAS | ATLAS catalogs adversarial AI techniques relevant to prompt and model abuse. |
Use AI RMF to inventory LLM failure modes, assign owners, and track mitigations across the lifecycle.
Related resources from NHI Mgmt Group
- How should organisations respond when an LLM passes broad safety tests but fails a specific use case?
- What do teams get wrong when choosing an LLM for a specific task?
- How should security teams use LLM-based identity risk scoring in production?
- Should organisations use new AI-specific identity standards or existing ones?
