What breaks when AppSec relies only on LLM-based analysis?

Why This Matters for Security Teams

When AppSec relies only on LLM-based analysis, the main failure is not simply that the model can be wrong. The larger issue is that its output is probabilistic, context-sensitive, and difficult to audit against a stable control baseline. That makes it risky for vulnerability triage, secure code review, policy enforcement, and evidence collection. The NIST AI Risk Management Framework is useful here because it treats AI use as a managed risk problem, not a substitute for engineering validation.

Security teams often assume an LLM can accelerate review without changing assurance requirements. In practice, it can surface plausible explanations for issues that do not exist, miss patterns it has not generalised well, or apply inconsistent reasoning to similar code paths. That creates friction with developers, but it also weakens governance because reviewers can no longer demonstrate why one finding was accepted and another was escalated. Current guidance suggests LLM output should be treated as advisory unless it is anchored to deterministic checks, policy rules, and reproducible test evidence. In practice, many security teams encounter this only after a false negative has already entered production or a flood of false positives has trained developers to ignore the queue.

How It Works in Practice

LLM-based AppSec analysis is strongest when used as a front-end summarisation or hypothesis generation layer, not as the final decision engine. It can help classify findings, explain code intent, suggest likely attack paths, and map code snippets to candidate weaknesses. It is much weaker when asked to prove absence of a flaw, because absence claims require repeatable validation. That is why mature programmes combine LLMs with static analysis, dependency scanning, secrets detection, policy-as-code, and targeted manual review.

A practical workflow usually looks like this: the model reviews code or a pull request, highlights suspicious constructs, and proposes a remediation path; then deterministic tools verify whether the condition exists; finally, a human validates risk and impact. This pattern aligns with the control emphasis in the NIST AI Risk Management Framework and the more implementation-focused NIST AI 600-1 Generative AI Profile, which both stress governance, measurement, and traceability.

• Use the LLM to prioritise review, not to certify code safety.
• Validate claims with deterministic scanners, tests, or policy checks.
• Record prompts, outputs, and reviewer decisions for auditability.
• Treat uncertain findings as candidates for deeper inspection, not closure.

This approach is especially important in CI/CD pipelines, where speed pressure can push teams to accept model output as evidence. It also matters in regulated environments where the security team must explain why a control passed or failed. These controls tend to break down when the codebase is highly dynamic, third-party libraries change frequently, and the model lacks reliable context about architecture, secrets handling, or runtime behaviour.

Common Variations and Edge Cases

Tighter review controls often increase cycle time, so organisations have to balance developer throughput against assurance quality. The tradeoff becomes sharper when teams try to use LLMs for broad code review, threat modelling, and policy decisions at the same time.

There is no universal standard for using LLMs as the primary AppSec analysis layer, and best practice is still evolving. Some teams use LLMs only for secure code explanations, while others allow them to draft remediation guidance but not to approve release gates. The difference usually comes down to whether the organisation can prove repeatability. Where regulated software, high-risk customer data, or privileged integrations are involved, a human plus deterministic evidence is still the safer model. Where the question intersects with agentic AI, the risk becomes wider because the system may also reason about tool use, code changes, or automated remediation, which introduces additional failure modes described in the OWASP Top 10 for Agentic Applications 2026 and the MITRE ATLAS adversarial AI threat matrix.

The edge case to watch is the feedback loop. If the LLM trains reviewers to trust its tone or confidence level, teams can accidentally promote output quality over factual accuracy. That is why guidance should stay grounded in validation, provenance, and reproducibility rather than fluent explanations.

Related resources from NHI Mgmt Group

• What breaks when appsec relies on code review alone?
• How should security teams use LLM-based identity risk scoring in production?
• What is the difference between static scanning and runtime analysis in AppSec?
• What breaks when AI security only relies on logging and alerting?