Notifications
Clear all
Topic starter
02/08/2026 12:19 pm
TL;DR: Domain grounding, not generic fluency, is becoming the decisive factor in security triage quality, according to Cogent’s benchmark comparison. Its security-trained agent reached 92.1% accuracy on root-cause mapping and 90.2% on multiple-choice cyber questions, outperforming GPT-5 and other lab models on CTI-Bench tasks designed to test real vulnerability research.
NHIMG editorial — based on content published by Cogent: Agent vs. General LLMs comparison of accuracy on vulnerability research
By the numbers:
- Root Cause Mapping: 92.1 percent accuracy, +6.0 points vs Sec-Gemini, +6.3 vs GPT-5 + Web Search.
- At 1,000 CVEs, that is roughly 60 and 63 fewer misclassifications.
Questions worth separating out
Q: How should security teams evaluate AI tools for code vulnerability discovery?
A: Use a fixed vulnerable corpus, repeat each test several times, and score the union of findings as well as the average single-pass result.
Q: Why do general LLMs struggle with cyber classification tasks?
A: General LLMs often recognise words and patterns without reliably mapping them to the correct security taxonomy.
Q: What breaks when AI coding agents are used as the final authority on vulnerabilities?
A: False positives, unstable results, and shallow reasoning become operational failures rather than harmless noise.
Practitioner guidance
- Benchmark AI triage on canonical weakness labels Test any security model against a labelled CVE or incident corpus and score exact-match performance on the classifications your team actually uses.
- Require source-backed reasoning paths Only trust outputs that show the evidence trail from source text to classification to recommendation.
- Separate classification from remediation approval Let AI help sort and summarise, but keep approval authority for fixes, exception handling, and risk acceptance with human reviewers.
What's in the full report
Cogent's full analysis covers the benchmark design and comparison mechanics this post intentionally leaves for the source:
- Exact CTI-Bench scoring approach for root-cause mapping and multiple-choice evaluation
- Side-by-side examples showing where general models chose the wrong CWE class
- System design details behind the security-trained agent's grounding and decision rules
- Limitations observed when public signals were sparse or contradictory
Vulnerability triage accuracy: what domain-trained AI changes for teams?
Explore further
02/08/2026 1:20 pm
Domain-trained AI is now a governance issue, not just a productivity feature. When security teams use AI for vulnerability triage, they are delegating a decision that affects prioritisation, escalation, and remediation sequencing. A model that understands security taxonomies can reduce noise, but a model that merely sounds confident can amplify triage errors at machine speed. For IAM and NHI teams, the same governance logic applies when models interpret credentials, access edges, or identity events. The practical conclusion is that AI outputs in security operations need classification assurance, not just language quality.
A question worth separating out:
Q: How do you know if a security AI model is actually helping?
A: Look for better downstream outcomes, not just higher benchmark scores. Useful signals include fewer mislabels, less analyst rework, faster triage, and more consistent remediation guidance. If the model cannot improve those operational measures, its output is probably descriptive rather than decision-grade.
👉 Read our full editorial: Domain-trained AI beats general LLMs on vulnerability triage
