Domain-trained AI beats general LLMs on vulnerability triage

At a glance

What this is: Cogent compares its security-trained agent with general LLMs on vulnerability research tasks and says the domain-trained system was more accurate on both root-cause mapping and security multiple-choice questions.

Why it matters: For IAM and security teams, the practical issue is not model fluency but whether AI can classify weaknesses correctly enough to drive remediation, prioritisation and reporting without compounding triage errors.

Context

Security triage fails when systems can describe a vulnerability but cannot reliably classify its root cause. In practice, that produces noisy prioritisation, generic remediation, and inconsistent reporting across vulnerability management, SOC, and application security workflows. The identity angle is indirect but real: if an AI assistant is helping analyse exposures, its output becomes part of the governance chain that determines what gets fixed first.

This article tests that problem with a benchmark setting rather than a product demo. The central question is whether domain-trained AI can map security signals more accurately than general-purpose LLMs, and the answer matters because security teams increasingly want AI to compress research time without weakening decision quality.

For practitioners, the relevant takeaway is that model choice should be judged by task fit, not by broad conversational ability. That is typical for high-volume triage environments, where precision on weakness classification has more operational value than general reasoning polish.

Key questions

Q: What fails when an AI triage tool cannot map vulnerabilities to the right weakness class?

A: The failure is not just a bad label. Wrong weakness classification sends remediation down the wrong path, weakens prioritisation, and distorts reporting. In practice, teams may patch symptoms instead of root causes, duplicate work across groups, and lose confidence in AI-generated guidance when the output sounds plausible but is operationally wrong.

Q: Why does domain-trained AI outperform general LLMs in vulnerability triage?

A: Domain-trained AI is anchored in security taxonomy, curated intelligence, and decision rules that reflect how vulnerabilities are actually analysed. General LLMs may reason well in language but often overfit to keywords or broad context. When the task is classification and prioritisation, structure and evidence usually matter more than conversational breadth.

Q: How do you know if AI triage is actually improving security outcomes?

A: Measure whether the triage decision can be reviewed, reversed, and tied back to concrete evidence. If false positives disappear but responders cannot explain why, the process has reduced noise without improving trust. Effective triage should shorten time to decision while preserving the ability to validate critical findings.

Q: Should security teams use one model for summarisation and triage decisions?

A: Usually not. Summary generation tolerates looser language, but triage decisions need high precision and traceable evidence. A better pattern is to use broader models for explanation and domain-trained systems for classification, prioritisation, and remediation recommendations. That separation reduces the chance that fluent but weak reasoning drives operational decisions.

Technical breakdown

Why root-cause mapping is harder than generic summarisation

Root-cause mapping asks an AI system to read a vulnerability description and assign the correct weakness class, not merely restate the issue. That is a classification problem grounded in taxonomies such as CWE, where the difference between a buffer overflow, hard-coded credential, or deserialization flaw changes remediation and validation. General LLMs often perform well on language but poorly on taxonomy precision because they anchor on surface terms instead of the technical mechanism. In security operations, that distinction matters because a wrong class can send engineers to the wrong fix path.

Practical implication: Measure AI triage tools on classification precision, not just summarisation quality.

Why authoritative context changes model output

Domain-trained systems improve because they are not reasoning from text alone. In this article, the agent uses verified vulnerability intelligence, normalised product and version data, and CWE-aware decision rules to narrow the candidate set before answering. That architecture reduces ambiguity when public sources conflict or when a CVE description contains overloaded security language. For triage teams, the technical lesson is that retrieval, normalisation and decision rules can matter as much as the base model. The model is only one layer in the control chain.

Practical implication: Feed AI triage workflows with curated vulnerability intelligence and explicit decision logic.

What benchmark scoring can and cannot prove

Exact-match benchmarks are useful because they show whether a system selected the right class or answer, not whether it sounded plausible. CTI-Bench uses canonical labels for root-cause mapping and exact-answer scoring for multiple-choice security questions, which makes it a good test of operational precision. But benchmarks still compress reality. Production triage also has to handle sparse evidence, vendor-specific product naming, and inconsistent disclosures. A strong score demonstrates competence under controlled conditions, not universal reliability across every incident workflow.

Practical implication: Validate domain AI on your own vulnerability distribution before using it in production triage.

NHI Mgmt Group analysis

Domain grounding is now a governance control, not just a model-quality preference. Security teams are not buying AI to produce prettier summaries, they are buying decision support that changes what gets fixed and when. If the model mislabels a weakness, the error propagates into remediation planning, reporting and program metrics. The practical conclusion is that triage accuracy has to be governed like any other security control outcome.

The named concept here is vulnerability triage fidelity. That is the degree to which an AI system preserves the correct weakness class, impact logic and remediation direction when converting raw vulnerability text into action. In this article, the domain-trained agent performs better because it starts from structured security intelligence rather than generic language priors. The implication is straightforward: fidelity, not fluency, is the relevant success criterion.

General-purpose LLMs will keep failing where security taxonomies are the task boundary. The issue is not that they cannot answer questions, but that they are not optimised for the distinction between symptom, root cause and mitigation class. That gap becomes more visible in vulnerability intake, threat intel enrichment and board reporting, where one wrong label can change the operational response. The practical conclusion is to reserve broad models for drafting and use domain-trained systems for classification.

Agentic AI in security workflows inherits the same identity and governance expectations as other machine actors. Once an AI system is allowed to classify vulnerabilities, recommend remediation or influence prioritisation, its outputs become part of the organisation’s control fabric. That means access, grounding, auditability and reviewability matter even when no human user is typing the final answer. The practical conclusion is to govern AI research agents as decisioning systems, not as chat interfaces.

This benchmark result supports a split between conversational AI and operational AI. Teams should not assume that a model good at explanation is also good at triage. In security operations, the control objective is consistency under pressure, and that requires model-task alignment, curated evidence and bounded outputs. The practical conclusion is to benchmark by workflow outcome, not by generic benchmark prestige.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• 67% of organisations still rely heavily on static credentials despite the risks they pose to agentic AI deployments, according to the 2026 Infrastructure Identity Survey.
• Read next: Top 10 Agentic AI Identity Issues

What this signals

Vulnerability triage fidelity: the practical challenge is preserving the correct weakness class when AI converts disclosure text into action. In security operations, that fidelity determines whether triage output drives the right fix or simply accelerates the wrong one.

The wider signal for practitioners is that AI quality needs to be judged at the workflow boundary, not the prompt boundary. A model that is persuasive but misclassifies weaknesses creates operational debt, while a model that is narrower but accurate can materially improve remediation discipline.

The same lesson is now showing up across agentic AI adoption more broadly, where 67% of organisations still rely heavily on static credentials despite the risks they pose to agentic AI deployments, according to the 2026 Infrastructure Identity Survey.

For practitioners

• Benchmark AI on weakness classification, not just language quality Test triage tools against real CVE descriptions, canonical CWE labels, and the types of ambiguous vendor disclosures your teams actually see.
• Separate summarisation from decision support Use general LLMs for drafting and explanation, but reserve domain-trained systems for root-cause mapping, prioritisation, and remediation guidance.
• Ground AI outputs in authoritative security sources Prioritise workflows that normalise product names, versions, and exploit evidence before the model produces an answer or recommendation.
• Audit misclassification impact across the remediation pipeline Track how wrong weakness labels affect deduping, change windows, mitigation selection, and programme reporting so you can quantify downstream cost.

Key takeaways

• The core problem is classification accuracy, not conversational quality, because weak weakness mapping changes remediation and reporting decisions.
• Benchmark-style evaluation matters when comparing security AI systems, but production value still depends on how well the model fits the real triage workflow.
• Teams should treat domain-trained AI as a decision-support control that needs evidence, auditability and ongoing validation, not as a generic chatbot.

Key terms

• Root-Cause Mapping: Root-cause mapping is the process of assigning a vulnerability to the underlying weakness class that explains why it exists. In security operations, the label matters because it determines remediation, validation and reporting logic, not just how the issue is described.
• Weakness Taxonomy: A structured list of named security defects that standardises how issues are described across teams and tools. In practice, it improves consistency between assessment, remediation, reporting, and compliance because the same weakness is identified the same way every time.
• Domain-Trained AI: Domain-trained AI is a model or agent tuned to perform a narrow operational task using specialised context, rules or evidence sources. In security, that usually means it is better at classification, triage and recommendation than a general model that only has broad language capability.
• Triage Fidelity: Triage fidelity is the degree to which an AI or analyst preserves the correct security meaning of an issue while turning raw signals into a decision. High fidelity means the output supports the right fix, the right priority and the right reporting outcome.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore nhimg.org for resources that connect identity governance to the broader security disciplines your programme depends on.