Why do LLMs complicate traditional application security assumptions in production systems?

Why This Matters for Security Teams

LLMs change the security model because the thing being protected is no longer just application code or API traffic. The model can interpret untrusted language, incorporate retrieved content, and influence downstream actions. That makes classic AppSec assumptions weaker: input may be semantically hostile without looking malformed, and a harmless-looking prompt can still steer tool use, data exposure, or business logic. Current guidance from the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 treats these as governance and application-risk problems, not just model quality issues.

That distinction matters in production. Traditional scanners may confirm that endpoints are patched, dependencies are current, and injection filters exist, yet still miss prompt injection, retrieval poisoning, indirect instruction handling, and unsafe action execution. Security teams also need to think about provenance of prompts, retrieval sources, and tool outputs because the model may act on content that was never intended as trusted input. The operational impact is often felt in access control, auditability, and response containment rather than in a single exploitable code flaw.

In practice, many security teams encounter LLM risk only after a model has already exposed data, taken an unsafe action, or been tricked through an indirect prompt path, rather than through intentional security testing.

How It Works in Practice

In production, an LLM usually sits inside a larger workflow: a user prompt enters an application, the application may retrieve documents, the model generates a response, and a separate tool or orchestration layer may decide whether to send an email, create a ticket, query a database, or call an API. That chain creates multiple trust boundaries. Traditional AppSec checks focus on syntax, authorization gates, and known software vulnerabilities, but LLM risk often emerges from semantics, context assembly, and decision delegation.

A practical control design starts with separating what the model can read from what it can cause to happen. The model should not inherit broad permissions simply because it is embedded in a business process. Security teams should classify inputs, constrain retrieval, validate outputs, and require explicit approval before high-impact actions. The NIST AI 600-1 Generative AI Profile is useful here because it translates AI risk into operational controls for governance, mapping, and monitoring.

• Apply allowlists for tools, commands, and data sources.
• Log prompts, retrieved context, tool calls, and final actions for review.
• Validate model output before it reaches users or automation.
• Isolate secrets, tokens, and sensitive records from the model context unless there is a clear business need.
• Test for prompt injection, indirect prompt injection, and retrieval poisoning as part of release readiness.

Security testing should also include adversarial scenarios that traditional AppSec would not cover, such as malicious instructions embedded in documents, web pages, or tickets that the model later retrieves. The MITRE ATLAS adversarial AI threat matrix is helpful for mapping these attack patterns, while the NIST Cybersecurity Framework 2.0 helps connect them to identify, protect, detect, respond, and recover outcomes. These controls tend to break down when LLMs are allowed to call multiple internal systems with inherited service credentials, because the blast radius becomes hard to predict and harder to contain.

Common Variations and Edge Cases

Tighter model governance often increases latency, review overhead, and integration cost, so organisations have to balance usability against the risk of autonomous or semi-autonomous action. That tradeoff becomes sharper as teams move from chat-style assistants to agentic workflows that can create records, move funds, or alter infrastructure.

There is no universal standard for every LLM deployment yet, so guidance should be scaled to the sensitivity of the workflow. A customer-facing summariser and an internal code-generation assistant do not carry the same risk as an agent that has access to production systems. In some environments, especially retrieval-augmented generation, the main weakness is not the model itself but the trust placed in the retrieved content. In others, the issue is overbroad privileges attached to the orchestration service account or insufficient separation between human approval and machine execution.

This is also where identity governance matters. If an AI agent can act on behalf of a person, or if it uses long-lived secrets to access systems, the organisation should treat that capability as a non-human identity concern with explicit ownership, review, and revocation. Best practice is evolving, but current guidance suggests that AI safety, application security, and identity controls should be designed together rather than managed as separate programs. The CSA MAESTRO agentic AI threat modeling framework is useful for identifying where those boundaries should sit.

Related resources from NHI Mgmt Group

• Why do AI agents complicate traditional application security assumptions?
• Why do AI systems complicate traditional data security controls?
• How should security teams implement application detection and response in production systems?
• How should security teams limit the risk from AI agents that have access to production systems?