Domain-trained AI beats general LLMs on vulnerability triage

At a glance

What this is: Cogent compared a security-trained agent with general LLMs on CTI-Bench tasks and reported higher accuracy on vulnerability root-cause mapping and technical security questions.

Why it matters: For IAM, NHI, and security operations teams, the finding matters because misclassification in intake, weak grounding, and generic guidance can slow remediation and distort risk decisions.

By the numbers:

• Root Cause Mapping: 92.1 percent accuracy, +6.0 points vs Sec-Gemini, +6.3 vs GPT-5 + Web Search.
• Multiple Choice Questions: 90.2 percent accuracy, +3.9 points vs Sec-Gemini, +2.1 vs GPT-5 + Web Search.
• 1, t 1,000 CVEs, that is roughly 60 and 63 fewer misclassifications.
• 2, t 2,500 questions, the reductions are about 98 and 53.

👉 Read Cogent's comparison of its security-trained agent against general LLMs on vulnerability research

Context

Vulnerability triage depends on correctly mapping symptoms to root cause, not just summarising text. In this article, the primary issue is the gap between generic language models and domain-trained systems when they are asked to classify CVEs, interpret exploit signals, and produce remediation advice. For security teams, the operational question is whether AI is reducing analyst workload or simply accelerating plausible but wrong conclusions.

That distinction matters across IAM and NHI programmes as well, because the same pattern shows up whenever models are allowed to reason over credentials, access paths, or security tickets without sufficient domain grounding. A system that can explain a vulnerability but not identify the weakness class can still drive poor remediation choices. The article frames a typical enterprise problem, not an edge case: accuracy falls when the model lacks the security context needed to reason precisely.

Key questions

Q: How should security teams evaluate AI tools for code vulnerability discovery?

A: Use a fixed vulnerable corpus, repeat each test several times, and score the union of findings as well as the average single-pass result. That approach reveals variance, missed paths, and cost efficiency. A model that looks weak once may be useful when pooled, while a headline score can hide unreliable performance.

Q: Why do general LLMs struggle with cyber classification tasks?

A: General LLMs often recognise words and patterns without reliably mapping them to the correct security taxonomy. That leads to plausible but incorrect labels, especially when the issue involves nuanced distinctions such as default credentials versus hard-coded secrets or buffer overflow versus broader memory corruption. Security work needs canonical reasoning, not fluent approximation.

Q: What breaks when AI coding agents are used as the final authority on vulnerabilities?

A: False positives, unstable results, and shallow reasoning become operational failures rather than harmless noise. Teams can waste time on non-issues, miss real defects, and lose confidence in their review process. AI should surface candidates, but a deterministic scanner or human reviewer must confirm the exploit path before remediation is prioritised.

Q: How do you know if a security AI model is actually helping?

A: Look for better downstream outcomes, not just higher benchmark scores. Useful signals include fewer mislabels, less analyst rework, faster triage, and more consistent remediation guidance. If the model cannot improve those operational measures, its output is probably descriptive rather than decision-grade.

Technical breakdown

Why root-cause mapping fails in general LLMs

Root-cause mapping asks a model to convert a vulnerability description into the correct weakness class. That is harder than summarisation because the model must distinguish the symptom from the underlying defect. General LLMs often latch onto keywords, surrounding context, or the most common weakness pattern in the corpus, which creates false precision. In security operations, that means the model may label a buffer overflow as a broader memory issue or treat hard-coded credentials as default credentials. The technical problem is not language generation, but weak canonical reasoning over security taxonomies.

Practical implication: validate AI-assisted triage against canonical weakness classes before it influences prioritisation or remediation.

How domain grounding changes cyber reasoning

Domain grounding combines authoritative sources, normalised product and version data, and explicit decision rules. In the article’s benchmark design, the security-trained agent used VulnCheck-backed intelligence and CWE-aware reasoning to select the most precise label before suggesting actions. That structure matters because cyber questions are rarely answered well by recall alone. A model needs to connect named products, exploit signals, and technical indicators to the correct taxonomy entry. Without that layer, the output can sound competent while still misclassifying the issue in ways that affect fix selection and validation.

Practical implication: require source-backed reasoning and taxonomy mapping in any AI workflow that classifies vulnerabilities or incident data.

Why accuracy gaps compound in operational workflows

A few percentage points of accuracy sound modest until they are applied at scale. In high-volume intake, each wrong label creates rework, delays prioritisation, and can send teams toward the wrong fix path. That is especially true when tickets, vuln feeds, and board reporting all depend on the same classification layer. The article’s own examples show why precision matters: the remediation for a command-injection issue is not the same as the remediation for deserialisation or credential misuse. Accuracy is therefore a governance control, not just a model metric.

Practical implication: treat classification quality as an operational control and measure it against downstream remediation outcomes.

NHI Mgmt Group analysis

Domain-trained AI is now a governance issue, not just a productivity feature. When security teams use AI for vulnerability triage, they are delegating a decision that affects prioritisation, escalation, and remediation sequencing. A model that understands security taxonomies can reduce noise, but a model that merely sounds confident can amplify triage errors at machine speed. For IAM and NHI teams, the same governance logic applies when models interpret credentials, access edges, or identity events. The practical conclusion is that AI outputs in security operations need classification assurance, not just language quality.

Security classification is the real benchmark, because root-cause precision determines action. The article’s comparison shows that exact-match tasks expose weaknesses general models tend to hide in free-text answers. That is why a CWE label, a weakness category, or an identity control type is more operationally useful than a fluent explanation. In broader security programmes, this argues for tighter coupling between AI reasoning and canonical control vocabularies such as MITRE ATT&CK and NIST CSF. The practitioner takeaway is to optimise for decision fidelity, not narrative polish.

AI governance debt appears when teams deploy models faster than they can validate their reasoning boundaries. The article demonstrates that model choice alone is not enough; the system design around it determines whether outputs are trustworthy. That includes curated sources, normalised inputs, and guardrails for uncertainty. In identity-adjacent workflows, this is especially important because mistaken judgments about access, privilege, or ownership can cascade into bad approvals. The practical conclusion is to define what the model may classify, what humans must verify, and where confidence thresholds trigger escalation.

Security AI should be evaluated as an operational control surface, not a chat interface. The useful question is not whether the model can answer a question, but whether it can improve triage quality under real workload pressure. This article points toward a market shift where domain-trained systems are judged on measurable precision, repeatability, and provenance. That aligns with the direction of NIST AI Risk Management Framework thinking: measurable performance, traceability, and manageable failure modes. The practical conclusion is to demand evidence of classification accuracy on your own distributions before operational adoption.

What this signals

AI governance debt: organisations that deploy security AI before defining classification boundaries will accumulate operational risk faster than they reduce analyst load. The immediate signal is not whether the model can answer a question, but whether it can do so with provenance and repeatability. For teams evaluating agentic workflows, the control question is whether AI outputs can be audited like any other decision in the pipeline, especially when the answer influences access, remediation, or escalation.

The practical direction is to pair model evaluation with identity and access governance. Where AI systems touch privileged data, security teams should align controls to the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10. That combination helps separate useful automation from ungoverned decision-making.

For identity-adjacent programmes, the lesson is that classification quality and access quality are now linked. If a model can misclassify a weakness, it can also mis-handle ownership, privilege scope, or remediation context. Teams should prepare for a future where AI-assisted security operations are judged by evidence quality, not interface polish.

For practitioners

• Benchmark AI triage on canonical weakness labels Test any security model against a labelled CVE or incident corpus and score exact-match performance on the classifications your team actually uses. Do not accept generic summaries as proof of operational value. Use the results to decide where the model can assist analysts and where it must stop at recommendation support.
• Require source-backed reasoning paths Only trust outputs that show the evidence trail from source text to classification to recommendation. Domain grounding should include normalised product names, version handling, and citation quality so analysts can audit why a label was chosen.
• Separate classification from remediation approval Let AI help sort and summarise, but keep approval authority for fixes, exception handling, and risk acceptance with human reviewers. This is especially important where a wrong weakness class could send teams to the wrong patch, mitigation, or validation step.
• Measure downstream operational impact Track whether better model accuracy reduces ticket rework, shortens triage time, and improves remediation quality. If those metrics do not move, the model is not providing usable security value even if benchmark scores look strong.

Key takeaways

• Security AI only helps when it maps issues to the correct weakness class, because root-cause precision determines the fix.
• Benchmark gains matter operationally when they reduce misclassification, rework, and decision drift at intake scale.
• Teams should govern security AI as a decision control, with provenance, auditability, and human approval boundaries.

Key terms

• Root Cause Analysis: Root cause analysis is the process of identifying why a control failed, not just what failed. It examines design, operation, training, authority, configuration, and dependencies so management can distinguish a one-off error from a systemic issue that needs deeper remediation.
• Canonical CWE Label: A Canonical CWE Label is the benchmark or reference weakness classification treated as the correct answer for a vulnerability. It provides a stable ground truth for scoring models and helps teams compare findings consistently across products, sources, and analysts.
• Grounding: The process of tying an AI system to the organisation's actual data, terminology, and operating context so it can act with relevant awareness. For identity governance, grounding is not just about accuracy. It is about limiting the agent to the business reality it was approved to operate in.
• Decision Fidelity: Decision Fidelity is the degree to which a system’s output supports the correct operational choice, not just a plausible explanation. In security operations, high decision fidelity means the model’s classification leads to the right fix, the right priority, and the right escalation.

What's in the full report

Cogent's full analysis covers the benchmark design and comparison mechanics this post intentionally leaves for the source:

• Exact CTI-Bench scoring approach for root-cause mapping and multiple-choice evaluation
• Side-by-side examples showing where general models chose the wrong CWE class
• System design details behind the security-trained agent's grounding and decision rules
• Limitations observed when public signals were sparse or contradictory

👉 The full Cogent post covers benchmark design, score breakdowns, and failure-mode examples.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to broader security operations and governance decisions.