Notifications
Clear all
Topic starter
01/08/2026 10:49 pm
TL;DR: Real vulnerabilities were found in an evaluation of Claude Code and OpenAI Codex across 11 large Python web apps, according to Semgrep. Claude Code reported 46 issues at 14% TPR and 86% FPR, while Codex reported 21 at 18% TPR and 82% FPR, showing AI can assist AppSec triage but cannot replace semantic analysis or deterministic validation.
NHIMG editorial — based on content published by Semgrep: AI Coding Agents Find Vulnerabilities in Real Applications
By the numbers:
- Claude Code found 46 vulnerabilities (14% true positive rate – TPR, 86% false positive rate – FPR)
- Codex reported 21 vulnerabilities (18% TPR, 82% FPR)
- One application produced 3, 6, and then 11 distinct findings across three identical runs
Questions worth separating out
Q: What breaks when AI coding agents are used as the final authority on vulnerabilities?
A: False positives, unstable results, and shallow reasoning become operational failures rather than harmless noise.
Q: Why do AI coding agents struggle with injection-style vulnerabilities?
A: Injection findings often depend on tracing data across functions, files, sanitisation layers, and framework abstractions.
Q: How should security teams use AI-assisted code review safely?
A: Use it as a triage layer that accelerates first-pass detection, then require a separate validation step for findings that affect access control, authentication, secrets, or release gating.
Practitioner guidance
- Require evidence-backed triage for AI findings Only accept AI-reported vulnerabilities when the finding includes a clear source-to-sink trace, a reproducible trigger, and a reviewer-confirmed security impact.
- Benchmark repeatability before adoption Run the same prompt multiple times against the same codebase and measure variance in findings, severity, and file coverage.
- Gate AI output behind deterministic scanners Use traditional SAST, taint analysis, or code review checks to validate AI candidates before they reach remediation queues.
What's in the full report
Semgrep's full post covers the operational detail this post intentionally leaves for the source:
- The full methodology for prompting Claude Code and OpenAI Codex across 11 real Python web applications.
- Per-vulnerability-class breakdowns that show where AI was strongest and where it failed on injection-style flaws.
- Details of the non-determinism tests, including repeated runs on the same codebase and prompt.
- The broader research questions and benchmark design discussion behind the evaluation approach.
👉 Read Semgrep's analysis of AI coding agents finding vulnerabilities in real Python apps →
AI coding agents and vulnerability hunting: where do they break down?
Explore further
01/08/2026 10:57 pm
AI-assisted vulnerability hunting is now a governance problem, not just a tooling problem. The article makes clear that AI coding agents can find real issues, but the signal is buried under a very high false-positive rate. That shifts the discussion from capability to control design, because security teams need evidence thresholds, validation steps, and clear ownership before AI output enters production workflows. The practitioner conclusion is simple: treat AI findings as input, not authority.
A question worth separating out:
Q: How should organisations govern AI tools that analyse source code?
A: Set evidence thresholds, require secondary validation for high-impact findings, and define who owns the final decision. Governance should cover prompt scope, output format, reviewer sign-off, and auditability. That keeps AI in an advisory role and prevents noisy model output from becoming an unreviewed security decision.
👉 Read our full editorial: AI coding agents find real bugs, but noisy results limit trust
