Topic starter
01/08/2026 10:49 pm
TL;DR: Real vulnerabilities were found in an evaluation of Claude Code and OpenAI Codex across 11 large Python web apps, according to Semgrep. Claude Code reported 46 issues at 14% TPR and 86% FPR, while Codex reported 21 at 18% TPR and 82% FPR, showing AI can assist AppSec triage but cannot replace semantic analysis or deterministic validation.
Editorial analysis by NHI Mgmt Group, based on content published by Semgrep: “Finding vulnerabilities in modern web apps using Claude Code and OpenAI Codex”.
By the numbers:
- Claude Code found 46 vulnerabilities (14% TPR, 86% FPR) across the evaluated applications.
- Codex reported 21 vulnerabilities (18% TPR, 82% FPR) in the same evaluation.
- The researchers reviewed more than 400 security findings produced by the two agents.
Key questions
Q: How should security teams handle AI-generated vulnerability findings in the release pipeline?
A: Security teams should treat AI-generated findings as inputs, not decisions.
Q: Why do AI coding agents produce so many false positives on injection bugs?
A: Injection bugs depend on semantic reasoning across sources, sinks, sanitization, and framework behaviour.
Q: What are the signs that an AI-assisted security review workflow is not reliable enough?
A: The main warning signs are inconsistent findings across repeated runs, a large share of issues that cannot be reproduced, and major performance swings by vulnerability class.
Practitioner guidance
- Establish human verification gates for AI findings Require a reviewer to confirm exploitability, data flow, and framework context before any AI-reported issue becomes a remediation item.
- Benchmark AI scanners by bug class Track performance separately for IDOR, path traversal, SQL injection, SSRF, and XSS so noisy classes do not obscure useful ones.
- Measure repeatability across identical prompts Run the same prompt multiple times on the same codebase and compare issue overlap, severity mix, and missed findings.
Bottom line: AI coding agents can find real vulnerabilities in production-grade code, but their outputs are too noisy to trust without validation.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
01/08/2026 10:57 pm
AI coding agents are becoming a security control dependency, not just a productivity feature. Once teams let model output influence bug triage, review prioritisation, or code fixes, the question becomes governance of evidence quality, not novelty. Security teams need to decide where AI assistance ends and where deterministic validation begins. The practitioner conclusion is to treat model output as advisory until it is verified by reproducible controls.
A few things that frame the scale:
- Software supply chain attacks were projected to cost organisations $60 billion in 2025.
A question worth separating out:
Q: Should organisations use AI coding agents for vulnerability hunting or human review?
A: Use them for candidate generation and triage assistance, not as a replacement for human review. AI is useful when the task is to surface possible bugs quickly, but humans and deterministic tools are still needed to confirm exploitability, interpret edge cases, and avoid acting on noisy or incorrect results.
👉 Read our full editorial: AI coding agents find real bugs, but noisy results limit trust