Why do LLMs create false positives in security testing?

They are optimised to produce the most plausible explanation, not to confirm reality. That means they can infer a weakness from patterns that resemble known exploits even when the target never behaves that way. False positives rise when teams confuse pattern recognition with evidence and let model confidence substitute for system verification.

Why This Matters for Security Teams

LLM-driven security testing can be useful for triage, but it becomes risky when teams treat the model’s output as evidence instead of hypothesis generation. A language model can mirror exploit language, associate a pattern with a vulnerability class, and still be wrong about the target’s actual behaviour. That is why false positives often appear credible: they are linguistically well formed, technically familiar, and easy to overtrust. NHI Management Group recommends anchoring review processes in validated observations and control evidence, not model fluency. The NIST AI Risk Management Framework is useful here because it frames AI outputs as risk-bearing artefacts that need governance, measurement, and human accountability.

The practical impact is not just noisy reporting. False positives can distort remediation priorities, waste analyst time, and create a false sense of coverage if teams believe a model has “found” what is only a pattern match. In adversarial environments, this matters even more because attackers can provoke models into overcalling weaknesses by feeding them misleading context or incomplete artefacts. In practice, many security teams encounter this only after a false positive has already been escalated into a ticket, not through intentional validation design.

How It Works in Practice

LLMs generate likely continuations based on training patterns and the prompt context, so they can infer a vulnerability from fragments that resemble known attack paths. In security testing, that becomes a problem when the model sees partial indicators such as endpoint names, API patterns, error strings, or policy snippets and then “fills in” the missing evidence. The model is often good at explaining why something might be exploitable, but not at proving that the exploit path exists in the live system.

Good practice is to use the model as a prioritisation layer, then require technical verification before a finding is accepted. That usually means:

• Checking the claim against source artefacts such as code, logs, configuration, and packet captures.
• Separating suspected issues from confirmed findings in the workflow and ticketing taxonomy.
• Using reproducible test cases so a human or automated scanner can confirm the issue independently.
• Recording model prompts and output context so reviewers can understand why the claim emerged.
• Applying change control when the model is used inside agentic workflows that can trigger actions.

This aligns with guidance in the OWASP Agentic AI Top 10 and the MITRE ATLAS adversarial AI threat matrix, both of which emphasise prompt manipulation, misleading inputs, and downstream misuse of model outputs. It also fits the operational discipline expected by the NIST AI 600-1 Generative AI Profile, which adds generative-AI-specific risk controls to the broader risk management model.

These controls tend to break down when the model is allowed to auto-triage findings in fast-moving environments where no separate verification step exists and analysts trust the output because the explanation sounds precise.

Common Variations and Edge Cases

Tighter verification often increases analyst workload and slows initial triage, requiring organisations to balance speed against evidential quality. That tradeoff is real, especially in vulnerability management, purple team exercises, and agentic AI security testing where teams want rapid coverage but cannot afford noisy findings.

There is no universal standard for how much confidence a model output should carry in a security workflow. Current guidance suggests treating confidence as advisory rather than authoritative, and making the burden of proof higher when the model is evaluating high-impact systems, production identities, or privileged paths. If the test involves credentials, session handling, or delegated access, the identity layer matters too: weak identity evidence can make a harmless lookup pattern look like an access-control flaw.

Edge cases also appear when the model is exposed to adversarial prompts, synthetic logs, or red-team content designed to resemble exploitation. In those cases, false positives may spike because the model is doing exactly what it was trained to do: infer the most probable next explanation. The safest response is to combine model-assisted analysis with deterministic scanners, manual inspection, and control validation against authoritative baselines such as NIST SP 800-53 Rev 5 Security and Privacy Controls and, where identity assertions are involved, NIST SP 800-63 Digital Identity Guidelines.

For teams building agentic review pipelines, the main lesson is simple: confidence is not confirmation, and the output should be logged as an assessment until evidence proves otherwise.

Related resources from NHI Mgmt Group

• When does static testing create a false sense of security?
• Why do failed auxiliary signals create false positives in real-time security systems?
• Why do false positives create security and compliance problems in KYC?
• Why do false positives create governance risk in application security?