What breaks when application security testing does not cover LLM-specific threats?

Why This Matters for Security Teams

When application security testing ignores LLM-specific threats, the assurance gap is not theoretical. Teams may still pass standard authentication, authorization, and input validation checks while missing model behaviour risks such as prompt injection, context abuse, tool manipulation, and unintended data exposure. That means a “secure” application can still be steered into unsafe actions or disclosure through prompts and retrieval paths. NIST’s NIST AI Risk Management Framework treats these issues as governance and lifecycle concerns, not just runtime defects.

The practical impact is broad. LLM features often sit inside workflows that touch customer data, internal knowledge bases, ticketing systems, code repositories, and agent tools. If testing only exercises classic web attack paths, the team learns very little about model outputs, instruction hierarchy, or whether the system can be induced to ignore policy and act on malicious content. Current guidance suggests that AI security testing must cover both the model interaction layer and the surrounding application controls. In practice, many security teams encounter these failures only after the model has already exposed data or executed an unsafe tool call, rather than through intentional pre-production validation.

How It Works in Practice

Effective testing for LLM-enabled applications needs to combine conventional application security checks with AI-specific abuse cases. The goal is to verify not only whether the app is reachable, but whether the model can be manipulated through prompts, retrieved content, or downstream tool instructions. The OWASP Agentic AI Top 10 and MITRE ATLAS adversarial AI threat matrix are useful references for translating those risks into test cases.

• Probe for prompt injection in user input, retrieved documents, and tool responses.
• Check whether the model can leak system prompts, secrets, or hidden retrieval content.
• Validate tool permissions so the agent cannot overreach its intended scope.
• Test input and output filters for bypasses, especially where content is reformatted by the model.
• Confirm that logs, traces, and prompt histories do not create a new sensitive-data store.

For regulated or high-risk use cases, testing should also verify provenance and configuration integrity across the AI supply chain, including model versions, system prompts, retrieval sources, and guardrail settings. The NIST AI 600-1 Generative AI Profile is helpful where teams need a more specific control lens for generative systems, while CSA MAESTRO agentic AI threat modeling framework supports scenarios where autonomous actions and tool use increase blast radius.

These controls tend to break down when LLM features are bolted onto legacy applications without a dedicated test harness, because the security team cannot safely simulate realistic prompt, retrieval, and agent execution chains.

Common Variations and Edge Cases

Tighter AI-specific testing often increases build and release overhead, requiring organisations to balance coverage against delivery speed and the current maturity of their AI controls. That tradeoff is real, especially where the model is embedded in a fast-moving product or where third-party components change frequently. Best practice is evolving, and there is no universal standard for this yet, particularly for agentic systems that can take actions across multiple services.

Some environments need deeper focus on sensitive-data handling, while others need stronger adversarial prompt testing or human approval gates for tool use. For example, customer support assistants may mainly need leakage and jailbreak testing, while internal coding agents need stronger checks on repository access, command execution, and secret retrieval. The Anthropic report on the first AI-orchestrated cyber espionage campaign shows why this matters operationally: once an agent can chain tasks, the testing problem becomes about abuse paths, not just isolated prompts. Teams should also monitor CISA cyber threat advisories for emerging attack patterns that may influence test cases.

The hardest edge case is hybrid systems that mix deterministic business logic with probabilistic model outputs. In those environments, the question is not whether the model is “accurate” in a general sense, but whether unsafe outputs can trigger privileged workflows, confuse downstream automation, or bypass human review.

Related resources from NHI Mgmt Group

• What breaks when LLM security is enforced only in the application layer?
• What breaks when application security testing is not tied to SEC disclosure readiness?
• What breaks when application security testing stops at isolated findings?
• What breaks when security testing does not cover the full attack surface?