Notifications
Clear all
Topic starter
02/08/2026 11:27 am
TL;DR: After benchmarking more than 2,000 findings, AI Assistant now matches internal security researchers on true positives in 96% of cases while still remaining conservative on false positives, according to Semgrep. That level of performance makes AI triage a governance issue, not just a workflow improvement, because teams must decide where machine judgment ends and human review begins.
NHIMG editorial — based on content published by Semgrep: AI triage accuracy and how Assistant reaches 96% true-positive agreement
By the numbers:
- Semgrep says Assistant agreed with security research triage decisions on true positives 96% of the time.
- When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, and as quickly as 9 minutes in some cases.
Questions worth separating out
Q: How should security teams govern AI agents that triage code vulnerabilities?
A: Security teams should treat code-triaging AI agents as governed production services, not prompts with a UI.
Q: How do you know if AI triage is actually improving security outcomes?
A: Measure whether the triage decision can be reviewed, reversed, and tied back to concrete evidence.
Q: What do security teams get wrong about low false positive rates?
A: They often assume a lower false positive rate means better security, but it may just mean the scanner is missing more issues.
Practitioner guidance
- Benchmark AI triage against real researcher decisions Use a labelled finding set that reflects your own languages, frameworks, and complexity profile, then measure true-positive and false-positive agreement separately.
- Set explicit suppression thresholds for low-confidence findings Define which classes of findings an assistant may suppress, which require human review, and which must always surface to developers.
- Preserve a human-auditable backlog for every AI verdict Keep suppressed findings visible to AppSec engineers with the model’s explanation, the original rule context, and the dataflow evidence used for the verdict.
What's in the full article
Semgrep's full analysis covers the operational detail this post intentionally leaves for the source:
- The benchmarking workflow used to compare assistant verdicts against security researchers across more than 2,000 findings
- Prompting and context design choices that changed model behaviour, including the role of rule confidence values
- The developer workflow mechanics for PR comments, Jira ticketing, and backlog tagging when findings are classified as true or false positives
- The trade-off analysis behind prioritising false-negative reduction over more aggressive false-positive suppression
👉 Read Semgrep's analysis of AI-assisted triage accuracy for AppSec findings →
AI triage for AppSec findings: can teams trust the verdicts?
Explore further
02/08/2026 12:43 pm
AI triage is becoming a governance control, not just an efficiency feature. Once an assistant decides which findings developers see, it is operating inside the security decision chain, not outside it. That means accuracy thresholds, confidence scoring, and override rights are governance issues as much as technical ones. For practitioners, the question is whether AI can reduce noise without becoming an unreviewed policy engine.
A question worth separating out:
Q: Who remains accountable when an AI assistant filters a security finding?
A: The organisation remains accountable, not the model. AppSec and security leadership must own the policy, the thresholds, and the review process that permit suppression. If the assistant hides a real issue, that is a governance failure in control design, not an excuse to defer responsibility to automation.
👉 Read our full editorial: AI triage accuracy changes AppSec governance more than noise reduction
