AI coding agents find real bugs, but noisy results limit trust

At a glance

What this is: Semgrep tested two AI coding agents on 11 real Python web applications and found they can identify genuine vulnerabilities, but their output is dominated by false positives and run-to-run inconsistency.

Why it matters: For IAM, NHI, and security teams, the important lesson is that AI-assisted review still needs governance, validation, and controls around trust, because noisy detection can create more triage burden than security value.

By the numbers:

• Claude Code found 46 vulnerabilities (14% true positive rate – TPR, 86% false positive rate – FPR)
• Codex reported 21 vulnerabilities (18% TPR, 82% FPR)
• One application produced 3, 6, and then 11 distinct findings across three identical runs
• The team evaluated 11 large, Python based, actively maintained open-source projects

👉 Read Semgrep's analysis of AI coding agents finding vulnerabilities in real Python apps

Context

AI coding agents are being used as vulnerability hunters before most organisations have decided how much trust to place in their outputs. In this case, the core problem is not whether the tools can find anything at all, but whether they can do so consistently enough to support real security operations.

For application security programmes, the governance issue is familiar: a tool that generates many plausible findings can still fail if it cannot distinguish evidence from noise. That matters to IAM and NHI teams too, because the same trust problem appears whenever automated systems are allowed to recommend, change, or validate security controls without strong verification boundaries.

Key questions

Q: What breaks when AI coding agents are used as the final authority on vulnerabilities?

A: False positives, unstable results, and shallow reasoning become operational failures rather than harmless noise. Teams can waste time on non-issues, miss real defects, and lose confidence in their review process. AI should surface candidates, but a deterministic scanner or human reviewer must confirm the exploit path before remediation is prioritised.

Q: Why do AI coding agents struggle with injection-style vulnerabilities?

A: Injection findings often depend on tracing data across functions, files, sanitisation layers, and framework abstractions. AI models can recognise suspicious patterns, but they frequently lose the full execution path. That makes them better at suggesting where to look than proving that SQL injection, XSS, or SSRF is actually present.

Q: How should security teams use AI-assisted code review safely?

A: Use it as a triage layer that accelerates first-pass detection, then require a separate validation step for findings that affect access control, authentication, secrets, or release gating. The safest pattern is hybrid review, where deterministic analysis and human judgement backstop the model’s reasoning.

Q: How should organisations govern AI tools that analyse source code?

A: Set evidence thresholds, require secondary validation for high-impact findings, and define who owns the final decision. Governance should cover prompt scope, output format, reviewer sign-off, and auditability. That keeps AI in an advisory role and prevents noisy model output from becoming an unreviewed security decision.

Technical breakdown

Why AI coding agents find real vulnerabilities but miss the full flow

AI coding agents are good at pattern recognition, especially when a vulnerability has a familiar shape such as an obvious IDOR or a likely injection point. They struggle when a finding depends on inter-procedural taint tracking, sanitisation logic, or a data flow that crosses files, frameworks, and abstractions. In practice, that means the model may see the right local signal but miss the end-to-end exploit path. The article’s results align with a broader AppSec reality: contextual reasoning helps, but semantic proof still requires deterministic analysis or human validation.

Practical implication: use AI to surface candidate issues, then require traceable data-flow evidence before treating a finding as real.

Why non-determinism is a security governance problem

Non-determinism means the same prompt and codebase can produce different findings on different runs. That is not just an engineering annoyance. It breaks comparability, complicates benchmarking, and makes it harder to tell whether a model improved or simply changed its guess. For security teams, inconsistency also creates operational risk because triage, escalation, and remediation planning all depend on stable evidence. Where a tool cannot reproduce its own conclusions, it should not be the final authority on vulnerability state.

Practical implication: benchmark AI review tools with repeated runs and compare result stability, not just headline accuracy.

Why SARIF output quality matters in AI-assisted AppSec

The article shows that tool usefulness is partly determined by the quality of the output format, not only by detection. If a model cannot emit valid SARIF reliably, downstream pipelines lose machine readability, deduplication, and ticketing automation. That is especially important in programmes that use AI findings to feed scanners, code review workflows, or remediation queues. AI-assisted security review is therefore an integration problem as much as a detection problem. Without reliable structured output, the value of the analysis collapses before it reaches operations.

Practical implication: reject AI security tools that cannot produce valid, consistently structured findings for your existing workflow.

Threat narrative

Attacker objective: The practical objective of the failure pattern is not exfiltration, but to mislead security operations with plausible yet unreliable vulnerability findings.

1. Entry occurs when the model is prompted to inspect real application code for vulnerabilities and begins generating candidate findings from surface patterns.
2. Credential access is not the central issue here, but the comparable failure mode is over-trust in tool output, where noisy findings gain credibility without sufficient proof.
3. Impact comes when teams treat inconsistent AI output as authoritative, increasing false triage, missed real issues, and wasted remediation effort.

NHI Mgmt Group analysis

AI-assisted vulnerability hunting is now a governance problem, not just a tooling problem. The article makes clear that AI coding agents can find real issues, but the signal is buried under a very high false-positive rate. That shifts the discussion from capability to control design, because security teams need evidence thresholds, validation steps, and clear ownership before AI output enters production workflows. The practitioner conclusion is simple: treat AI findings as input, not authority.

Detection non-determinism creates audit friction for AppSec and adjacent identity programmes. When the same prompt produces materially different findings, the programme loses a stable basis for triage, trend analysis, and change control. That matters beyond code review, because identity governance also depends on repeatable decisions around access, privilege, and policy exceptions. The right lens is operational integrity, not novelty, and that makes reproducibility a security requirement.

Semantic reasoning is the decisive boundary between useful assistance and unreliable automation. The article shows that these tools handle familiar patterns better than cross-file reasoning, sanitisation logic, and multi-step flows. That is the same class of problem that appears in agentic AI security and NHI governance, where a system may appear competent while still lacking enough context to safely act. The practitioner conclusion is to pair generative analysis with deterministic controls and explicit review gates.

Structured security output matters because downstream automation cannot consume ambiguity safely. Invalid SARIF is not a formatting footnote, it is an operational blocker. If a security programme intends to route AI findings into tickets, dashboards, or policy enforcement, then output integrity becomes part of the control surface. The field should stop asking whether AI can find bugs in the abstract and start asking whether it can support governed workflows end to end. The practitioner conclusion is to evaluate integration quality as rigorously as model performance.

Context engineering is emerging as the real differentiator in AI security workflows. The article’s results suggest that a model alone is insufficient, but an agent wrapped in the right code context, validation steps, and task decomposition can become materially more useful. That pattern is familiar in NHI and agentic AI security too: the system around the model matters as much as the model itself. The practitioner conclusion is to invest in scaffolding, not just prompts.

What this signals

Confidence in AI security tooling will remain low until review pipelines become more deterministic. The practical lesson for AppSec and platform teams is that model quality alone will not be enough. Organisations should plan for hybrid workflows where AI accelerates discovery but deterministic tools and human validation still own the final call, especially for high-impact findings.

Context engineering is becoming a control plane concern, not just a prompt-tuning exercise. When security value depends on feeding the model the right files, paths, and dependencies, the surrounding workflow becomes part of the defence model. That is why the line between AI governance and software security is narrowing, and why programmes need explicit review standards before scaling usage.

AI-assisted review should be measured like any other security control: coverage, stability, and decision quality. If findings fluctuate wildly or cannot be consumed by downstream systems, the programme is not gaining resilience. Teams that want to use AI in AppSec should define success in operational terms, not just in model output volume.

For practitioners

• Require evidence-backed triage for AI findings Only accept AI-reported vulnerabilities when the finding includes a clear source-to-sink trace, a reproducible trigger, and a reviewer-confirmed security impact. This reduces false positives from plausible but incorrect suggestions and keeps analyst time focused on real defects.
• Benchmark repeatability before adoption Run the same prompt multiple times against the same codebase and measure variance in findings, severity, and file coverage. If the tool cannot reproduce its own output, use it as a helper for exploration, not as a control in the review pipeline.
• Gate AI output behind deterministic scanners Use traditional SAST, taint analysis, or code review checks to validate AI candidates before they reach remediation queues. The article’s results show that AI is strongest as a discovery layer, while deterministic tooling remains better at proving flow-based vulnerabilities.
• Normalize structured output requirements Reject tools that cannot emit valid SARIF or equivalent machine-readable findings consistently. Downstream automation depends on stable schemas for deduplication, reporting, and ticket creation, and malformed output breaks the workflow before triage even begins.
• Treat AI security review as a controlled workflow Define where human approval is mandatory, which classes of findings require secondary validation, and how exceptions are recorded. This is especially important when the model is used to assist in broader agentic or NHI-related security tasks, where context and trust boundaries are easy to blur.

Key takeaways

• AI coding agents can identify real vulnerabilities, but high false-positive rates make raw output unsafe as a final security decision.
• Run-to-run inconsistency is a governance problem because unstable findings cannot support reliable triage, audit, or remediation.
• The best operating model is hybrid: AI for discovery, deterministic tools for proof, and human review for the final decision.

Key terms

• True Positive Rate: True positive rate is the share of reported findings that are confirmed as real issues. In security tooling, it measures how often a model or scanner correctly identifies a vulnerability rather than flagging something harmless as suspicious.
• False positive closure rate: The share of alerts that are automatically identified as benign and closed with supporting evidence before reaching analyst queues. It is a useful SOC metric because it shows whether automation is reducing noise without hiding real threats.
• Non-determinism: Non-determinism means the same input can produce different outputs across repeated runs. In AI security review, that makes benchmarking, auditability, and remediation planning harder because the result is not stable enough to function as a reliable control signal.
• SARIF: SARIF is a machine-readable format for security analysis results. It allows scanners and review tools to pass findings into downstream systems consistently, which is essential when security output must feed tickets, dashboards, or automated remediation workflows.

What's in the full report

Semgrep's full post covers the operational detail this post intentionally leaves for the source:

• The full methodology for prompting Claude Code and OpenAI Codex across 11 real Python web applications.
• Per-vulnerability-class breakdowns that show where AI was strongest and where it failed on injection-style flaws.
• Details of the non-determinism tests, including repeated runs on the same codebase and prompt.
• The broader research questions and benchmark design discussion behind the evaluation approach.

👉 Semgrep's full post covers the benchmark design, model-by-model results, and non-determinism tests in detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and agentic AI identity. It helps practitioners build the controls and review discipline needed to govern automated security systems safely.