AI And LLM Security

Expanded Definition

AI and LLM security is the discipline of protecting model behaviour, prompts, training and retrieval data, and the surrounding application from confidentiality, integrity, availability, and governance failures. It includes both the base model and the orchestration layer that connects users, tools, and data sources.

The boundary that matters most is that this term is broader than model safety alone. Model safety focuses on harmful outputs and alignment, while AI and LLM security also covers prompt injection, data leakage, tool abuse, insecure connectors, and weak access controls around the system that hosts the model. In practice, the application layer often becomes the real control plane, because a well-governed model can still be misused through insecure prompts, retrieval paths, or agent workflows.

For a formal risk framing, NIST’s AI Risk Management Framework is useful because it treats AI harm as a lifecycle issue rather than a single-model problem.

Examples and Use Cases

AI and LLM security appears differently depending on whether the system is conversational, retrieval-augmented, or tool-enabled. The same controls are not equally effective in each design, which is why practitioners need to match the control surface to the deployment pattern.

• A customer support chatbot is checked for prompt injection so it does not reveal restricted policy text or internal records.
• A retrieval-augmented generation system is scoped so the model can only access approved documents, not the full enterprise knowledge base.
• An internal coding assistant is restricted from calling deployment or secrets-management tools without explicit human approval.
• A model evaluation pipeline is used to test whether jailbreaks, data extraction, or unsafe instruction following still occur after each release.
• An agentic workflow is reviewed for tool permissions, because autonomous execution changes the blast radius of a bad prompt or malicious input.

Where the system is agentic rather than purely generative, the distinction is important: the security question shifts from only “what did the model say?” to “what action did the system take?” That is why the OWASP Top 10 for Agentic Applications 2026 is more relevant for tool-using systems than for chat-only deployments.

Security Implications

The main failure mode in AI and LLM security is assuming the model itself is the only asset that needs protection. In reality, the prompt channel, retrieval sources, plugin calls, logging, and downstream automation can all become exposure points. A model can be technically accurate and still be operationally unsafe if it can be induced to reveal sensitive data or trigger an unintended workflow.

Common consequences include confidential information leakage, corrupted outputs that influence business decisions, unsafe tool execution, and silent policy bypass through prompt manipulation. These failures are often hard to spot because they may look like ordinary user interactions rather than obvious attacks. The observable symptom is usually not a crashed service but a trustworthy system producing unauthorised or unreviewed action.

Where agentic features are present, the blast radius increases because a single successful input may move from content generation into execution. That is why adversarial AI references such as MITRE ATLAS adversarial AI threat matrix are useful for understanding abuse patterns that target the model, its context, and its surrounding controls.

Domain and Governance Relevance

From a governance perspective, AI and LLM security is not only a technical hardening problem. It requires clear ownership for model risk, data access, prompt boundaries, evaluation, monitoring, and incident response. Organisations that treat AI as a standalone feature often miss the fact that the surrounding application, identity, and data controls determine most of the real exposure.

This is where broader AI governance and safety frameworks become useful. They help define accountability for training data, third-party services, human oversight, and change control across the lifecycle. For agentic systems, the governance question becomes sharper because autonomous tool use can turn a content issue into an operational event.

For regulated environments, this term also connects to privacy, recordkeeping, and acceptable-use obligations because AI outputs may be generated from personal, confidential, or regulated data. For that reason, the NIST AI 600-1 Generative AI Profile is a strong fit for generative deployments, while the OWASP Agentic AI Top 10 remains the more precise lens when the system can act, not just answer.

Risk and Threat Considerations

AI and LLM security is exposed to both misuse and adversarial manipulation. The material risk is that an attacker, insider, or careless user can shape inputs, context, or tool requests so the system leaks data, follows malicious instructions, or performs an unintended action.

Failure mechanism: The recognised mechanisms are prompt injection, indirect prompt injection through retrieved content, insecure tool invocation, data poisoning, and excessive trust in model output. These failures occur when the application treats model responses or external context as trustworthy without enough filtering, approval, or isolation.

Impact: Sensitive data can be disclosed, business logic can be bypassed, and autonomous workflows can be pushed into unsafe execution. In agentic deployments, the consequence is wider because compromise of the conversational layer can become compromise of the action layer.

Related resources from NHI Mgmt Group

• How should security teams handle trust assumptions in LLM and AI agent workflows?
• How should security teams test LLM fingerprinting in production AI agents?
• How should security teams govern LLM outputs in production AI applications?
• Why does LLM routing create more security risk even when it lowers AI costs?