How should security teams evaluate LLMs for triage decisions?

How to test an LLM for triage work, not just model quality?

An LLM used for triage should be evaluated as a decision system, not a generic language model. That means testing it on the same case types, evidence quality, and label set it will face in production, then checking whether it makes stable decisions when the context is incomplete, noisy, or ambiguous.

The first question is whether the test set mirrors the operational queue. If the model will help with phishing, account takeover, and network triage, each of those scenarios needs its own slice of data, because the evidence signals and acceptable escalation thresholds are different. A single blended benchmark often hides where the model is overconfident or too conservative.

Evaluation also needs to reflect how analysts actually work. A useful triage model should be judged on whether it supports the right next action, not only whether it matches a final label. That means including examples where the correct outcome is to escalate, hold, request more context, or defer to a human reviewer, because those are real decisions in a triage workflow.

It also helps to separate the evidence from the conclusion. For triage, the same alert can lead to different decisions depending on what is present in logs, identity signals, user history, or containment state. A strong evaluation therefore tests whether the model changes its recommendation for the right reasons when key facts are missing, rather than treating every partial case as if it were complete.

What failure modes matter most in triage evaluation?

One common failure mode is false confidence on underspecified cases. If the model routinely picks a clean label when the evidence is thin, it may look accurate in aggregate while still sending analysts down the wrong path on edge cases. Another failure mode is label drift, where the model uses the right words but applies them inconsistently across similar incidents.

Disagreement patterns are especially useful because they show whether the model is systematically confused in specific conditions. For example, if it struggles with account takeover when logs are partial, or treats network anomalies as user compromise, that is not random error, it is a workflow problem that can be corrected with better prompts, better labels, or narrower task scope.

Escalation quality matters as much as final accuracy. A triage system that occasionally misses a low-risk alert may still be usable if it escalates uncertain cases early and clearly. A system that is highly accurate on easy cases but evasive or inconsistent on borderline ones creates more operational risk than its headline score suggests.

What should security teams measure before trusting an LLM for triage?

Measure performance by case type, not just overall. Teams should compare precision, recall, escalation rate, and disagreement rates across phishing, account takeover, and network categories, then review where missing context changes the outcome. That tells you whether the model is robust across the queue or only strong on a narrow subset of examples.

It is also worth measuring decision stability. If small changes in wording, evidence ordering, or context length cause different triage labels, the model is too brittle for production use. For operational settings, consistency under realistic variation is often more important than a marginal gain in benchmark score.

Where the output affects containment or analyst prioritisation, review the model's justification quality as well. The explanation should point to the evidence that actually drove the decision, so a reviewer can tell whether the model is reasoning from incident signals or merely pattern-matching from surface features.

Risk and Threat Considerations

LLM triage systems can fail in ways that are operationally expensive even when they appear to perform well in aggregate. The main risk is overtrust, where analysts inherit a model's confidence without enough visibility into why it chose a label, especially in cases with partial evidence or ambiguous signals.

Failure mechanism: The model overfits to training patterns, collapses distinct incident types into one another, or treats incomplete evidence as sufficient to make a firm recommendation, which can suppress escalation or misroute incidents.

Impact: Mis-triaged phishing, account takeover, or network cases can delay containment, waste analyst time, and hide systematic blind spots that only appear under real queue conditions.

Practitioner Guidance

What to verify: Test the model on production-like examples, including partial evidence, conflicting signals, and cases where the right answer is uncertainty or escalation. If it only performs well on clean, fully described incidents, it is not ready for live triage.

Decision rule: Treat disagreement and escalation behaviour as first-class evaluation outputs. If the model is accurate on easy cases but unstable on borderline ones, narrow its role to recommendation support rather than final triage decisions.

Practitioner takeaway: The right standard is not whether the LLM sounds right, but whether it makes the same judgment a competent analyst would make when the evidence is messy, incomplete, and time-sensitive.

Related resources from NHI Mgmt Group

• How should security teams evaluate LLMs for cybersecurity use beyond benchmark scores?
• How should security teams evaluate adversarial robustness in machine learning models used for production decisions?
• How should security teams evaluate LLMs for enterprise workloads instead of relying on public benchmarks alone?
• How should security teams evaluate SOC-as-a-Service when they need deeper investigation rather than basic alert triage?