Why do general-purpose LLMs miss vulnerabilities that human testers still find?

They miss them because many real flaws are not pattern matches. Business logic failures, custom trust boundaries, and newly patched environments depend on current system behaviour, not on a familiar exploit signature. Human testers can adapt to those conditions, while a general-purpose model often defaults to the nearest known pattern.

Why This Matters for Security Teams

General-purpose LLMs can be useful for triage, summarisation, and broad hypothesis generation, but they are not dependable substitutes for hands-on vulnerability discovery. The gap matters because many defects are contextual: they emerge from business rules, environment state, chained permissions, or the way a target system behaves under pressure. That is exactly where human testers still outperform models. The NIST AI Risk Management Framework is clear that AI outputs need governance, validation, and human oversight rather than blind trust.

Security teams often underestimate how much current state matters. A model may recognise a familiar pattern such as injection, broken auth, or unsafe deserialisation, yet still miss the actual weakness if the exploit depends on a unique workflow, a stale policy, or a recently changed integration. In practice, many security teams encounter the real flaw only after the model has already produced a plausible but incomplete assessment, rather than through intentional validation of the system’s actual behaviour.

How It Works in Practice

Human testers adapt to what the system is doing right now. They probe trust boundaries, observe error handling, test alternate paths, and notice when the application violates its own rules. General-purpose LLMs usually operate by pattern completion, so they are strongest when the issue resembles examples seen during training or in the prompt. They are weaker when the weakness is novel, highly specific, or only visible after chaining several small behaviours together.

This is why effective vulnerability discovery usually combines automation with targeted human analysis. A model can help draft test ideas, map attack surfaces, or summarise logs, but a tester still needs to verify whether those ideas hold against the live target. Current guidance from OWASP Agentic AI Top 10 and the NIST AI 600-1 Generative AI Profile both point toward treating model output as advisory, not authoritative.

In practical testing, the workflow often looks like this:

• Use the LLM to generate a broad hypothesis list from code, API specs, or user journeys.
• Have a human tester prioritise edge cases that depend on state, sequencing, or privilege boundaries.
• Validate findings against the live environment, not just static descriptions.
• Compare model suggestions with telemetry, logs, and error messages to catch false confidence.
• Retest after patching, because a model may not understand whether the environment is now partially fixed or newly exposed.

This approach is especially important for agentic workflows, where autonomous tools may amplify small mistakes into real impact. Resources such as the MITRE ATLAS adversarial AI threat matrix and the CSA MAESTRO agentic AI threat modeling framework are useful for framing how AI systems fail under manipulation, but they do not replace manual verification of application logic. These controls tend to break down when the target has custom state transitions, hidden approvals, or recently deployed changes because the model has no reliable way to infer the live operational context.

Common Variations and Edge Cases

Tighter model-driven testing often increases review overhead, requiring organisations to balance speed against confidence. That tradeoff becomes sharper when teams use LLMs for internal applications, complex SaaS integrations, or heavily customised workflows. In those environments, the model may identify generic weaknesses while missing the specific condition that makes the issue exploitable.

Best practice is evolving, but current guidance suggests treating LLMs as assistants for breadth, not as final judges of exploitability. Human testers are still needed when the question is whether a control actually works under real operational conditions. This is particularly true for business logic flaws, where the problem may be perfectly valid input that still produces the wrong outcome, and for post-patch validation, where the system can behave differently from what documentation or prior scans suggest.

There is also an important identity and agentic AI intersection. If an AI agent can call tools, inherit permissions, or act on behalf of a user, the testing problem shifts from “can the model recognise a vulnerability” to “can the environment constrain what the agent is allowed to do.” That is why practitioner teams increasingly pair application testing with identity-aware control review, especially when an agent can trigger actions that affect secrets, access rights, or downstream systems. In those cases, the hard part is often not detection but understanding whether authority has been granted too broadly.

Related resources from NHI Mgmt Group

• Why do raw LLMs miss so many vulnerabilities in code review?
• Why do AI-generated security summaries still need human governance?
• Why do traditional access reviews miss non-human identity risk?
• Who is accountable when an inactive non-human identity is still present after business use has ended?