Benchmarking LLMs for security triage shows model choice is use-case specific

At a glance

What this is: This benchmark compares leading LLMs on 163 real security triage decisions and finds that the best model changes by use case.

Why it matters: It matters to IAM, SOC, and security automation teams because model selection for phishing, account takeover, and network triage affects false positives, escalation quality, and where human review still has to stay in the loop.

By the numbers:

• Legion AI evaluated 163 cleaned triage decisions from real-world security investigations after removing mock runs and incomplete workflows.
• The benchmark found Gemini 3 Pro performed best overall at 74.8%, with GPT-4.1 and Opus 4.5 tied for second.
• False positives made up 91 of the 163 decision labels in the filtered dataset.

👉 Read Legion AI's benchmark of LLM performance across security triage use cases

Context

Security triage is only as reliable as the decision context the model receives. In SOC and identity workflows, an LLM is often choosing between benign, malicious, requires review, or escalation based on incomplete investigation data, which means performance depends on the task, the workflow, and the quality of the supporting evidence. For identity security teams, that is especially relevant where phishing, account takeover, and alert enrichment intersect with IAM and access decisions.

Legion AI's benchmark is useful because it tests LLMs against actual investigation decisions rather than synthetic prompts. The broader lesson is that model benchmarking in security operations should measure workflow fit, not abstract intelligence, and that appears to be the typical failure mode when teams treat one model as equally capable across all triage categories.

Key questions

Q: How should security teams evaluate LLMs for triage decisions?

A: Evaluate LLMs against the exact workflows, evidence types, and decision labels they will see in production. Use separate test sets for phishing, account takeover, and network cases, then measure not just accuracy but disagreement patterns, escalation quality, and how often missing context changes the outcome.

Q: Why do different LLMs create different security risks for the same application?

A: Different models vary in training data, alignment, context capacity, and resistance to prompt manipulation, so the same workflow can behave differently across systems. That matters when models handle sensitive data, tool calls, or autonomous actions. Security teams should treat model choice as part of the control design, not just a performance decision, because risk depends on behavior as much as capability.

Q: What makes automated security triage fail in practice?

A: It fails when the workflow feeds the model incomplete or noisy evidence, when decision options are poorly defined, or when analysts assume one model fits every queue. In those cases, the model is often responding to system design problems rather than showing a pure reasoning failure.

Q: How do teams know if LLM triage is actually working?

A: Teams should look for stable decision quality across queues, low rates of avoidable false positives, and consistent agreement between analyst judgement and model recommendations. If performance changes sharply by use case, the model is not universally reliable and routing rules need to be tightened.

Technical breakdown

How triage decision nodes work in security workflows

A triage decision node is a controlled point in an investigation workflow where the model chooses among predefined outcomes after receiving the evidence gathered so far. That makes the benchmark closer to production reality than a simple classification test, because the model is not deciding in isolation. It is reasoning over a workflow summary, tool outputs, customer-specific knowledge, and a constrained set of decision options. The quality of the decision therefore depends on both the model and the information pipeline that feeds it. In security operations, this is the difference between a model that can label a case and a model that can support a repeatable investigation process.

Practical implication: benchmark models against the exact decision points and evidence types used in production workflows.

Why phishing, account takeover, and network cases need different models

The benchmark shows that model strength is use-case dependent because each category asks the model to weight evidence differently. Phishing often depends on message content, sender trust, and external indicators. Account takeover is more dependent on identity signals, abnormal login patterns, and credential context. Network investigations rely more on infrastructure evidence, telemetry, and correlated security tooling. A model can be strong in one category and only average in another because the reasoning patterns are not the same. For IAM and SOC teams, that means model selection is a workload-design problem, not a generic vendor comparison.

Practical implication: route different investigation types to the model that performs best for that evidence pattern.

Why workflow quality matters as much as model quality

The dataset was cleaned to remove mock runs, interrupted workflows, and cases with missing information because those failures distort evaluation. That is important: poor workflow inputs can make a capable model look weak, while good workflow design can make a mediocre model look stronger than it is. In security automation, the model is often only one control in a chain that includes enrichment, analyst context, and response thresholds. If any of those stages are incomplete, the apparent model error rate rises even when the underlying issue is process design. The lesson is that triage quality is a system property, not a model-only property.

Practical implication: validate data completeness and workflow integrity before drawing conclusions from model scores.

Threat narrative

Attacker objective: The practical objective is to influence or exploit automated triage so that real threats are deprioritised or benign activity is escalated into wasted analyst effort.

1. Entry occurs when a security workflow receives incomplete or noisy evidence and hands it to an LLM as if the case were well formed.
2. Escalation happens when the model overweights missing, misleading, or poorly structured context and selects the wrong triage label.
3. Impact is misclassification, which can create false positives, missed threats, or unnecessary analyst workload in security operations.

NHI Mgmt Group analysis

Model benchmarking in security operations is really workflow benchmarking. The headline score matters less than whether the model can make the right decision at the right point in a live investigation. In SOC and IAM-adjacent triage, evidence quality, decision options, and contextual enrichment drive outcomes as much as model capability. Practitioners should treat evaluation as a test of the whole operating model, not a contest between model names.

Security triage should be segmented by use case, not flattened into one accuracy number. Phishing, account takeover, and network investigations rely on different evidence and different risk thresholds, so one model can be best overall and still be wrong for a specific queue. That is a governance problem as much as a technical one, because service-level expectations, analyst review rules, and escalation thresholds need to reflect the category being handled. Practitioners should align model routing to the investigation type, not the procurement narrative.

False positives are a governance signal, not just a nuisance metric. When 91 of 163 labels are false positives in the filtered set, the real question is whether the workflow is producing enough signal for a model to make a reliable decision. That matters for identity teams because alert quality and access context are often tangled together in phishing and account takeover reviews. Practitioners should measure how much of the apparent model error is actually upstream workflow noise.

Decision isolation creates benchmarking blind spots. A model that receives only a partial workflow summary can look weaker than it would in a richer investigative context, which is a common failure mode in automated security operations. This is where identity-aware telemetry, case enrichment, and ownership of workflow data become critical. Practitioners should treat context assembly as a control surface, not an implementation detail.

AI-assisted triage will increasingly require identity governance of the workflow itself. When models select decisions, they are effectively acting inside a privileged operational path that can amplify both good and bad inputs. That creates a governance requirement around who can shape workflows, which data sources the model can see, and what review gates apply before action is taken. Practitioners should govern the triage path with the same discipline they apply to other high-impact decision systems.

What this signals

Decision quality in AI-assisted triage will increasingly depend on governance of the surrounding workflow. The model is only one element in the control path. If evidence assembly, decision taxonomy, or review thresholds are inconsistent, the resulting automation will look brittle even when the underlying model is strong. Practitioners should treat the case pipeline as a governed asset, not a background implementation detail.

Identity context is becoming part of the AI operations stack. In phishing and account takeover handling, the model's judgement depends on access patterns, authentication signals, and user-behaviour context. That creates a direct link between IAM telemetry, SOC triage, and AI decision quality. Teams that separate those functions too sharply will struggle to explain why one queue performs better than another.

Model selection is moving toward function-level fit, not platform-level preference. The operational question is no longer which model is best in general, but which model handles a specific queue with the least analyst rework and the most consistent escalation behaviour. That change will favour teams that can measure by workflow, not by marketing claim.

For practitioners

• Benchmark by queue, not by headline score Test phishing, account takeover, and network triage separately against your own investigation patterns, because the best model for one queue may underperform in another. Use the same decision labels, enrichment sources, and escalation rules that analysts actually use.
• Audit workflow completeness before model tuning Check whether missing steps, partial evidence, or interrupted workflows are inflating apparent model error. If the input pipeline is inconsistent, fix the enrichment and case assembly process before changing models.
• Route identity-heavy cases through stricter review gates Apply more conservative thresholds to phishing and account takeover cases where identity signals, access context, and user behaviour are tightly coupled. That helps reduce false confidence when the model is missing the identity context needed for a good decision.
• Measure disagreement patterns, not just accuracy Track where the model and analyst disagree, then classify whether the cause is bad reasoning, missing data, or the wrong decision taxonomy. That gives you a better view of whether the problem sits in the model, the workflow, or the underlying policy.

Key takeaways

• The benchmark shows that security triage performance is use-case specific, so one model cannot be assumed to fit every queue.
• Workflow completeness and evidence quality are part of triage accuracy, not just the model's internal reasoning.
• Teams should route phishing, account takeover, and network cases differently and measure disagreement patterns before scaling automation.

Key terms

• Triage Decision Node: A triage decision node is a fixed point in an investigation workflow where a model or analyst chooses from predefined outcomes after reviewing gathered evidence. It is important because it turns unstructured security work into a controlled, measurable decision process that can be benchmarked across cases and teams.
• Workflow Completeness: Workflow completeness describes whether an investigation has all required inputs, enrichment steps, and outputs available when a decision is made. In model evaluation, incomplete workflows can make a capable system appear unreliable, because the decision is being made on partial evidence rather than a true operational record.
• False Positive: A false positive is a scanner result that looks like a secret but is not actually sensitive. In secret governance, false positives matter because they consume analyst time, weaken trust in alerts, and can delay response to the findings that truly change exposure and access risk.
• Use-Case Segmentation: Use-case segmentation is the practice of evaluating and operating security models separately for different investigation categories such as phishing, account takeover, and network activity. It matters because the same model can behave differently depending on the evidence type, risk tolerance, and decision logic required by each queue.

What's in the full article

Legion AI's full benchmark covers the methodology and evaluation detail this post intentionally leaves for the source:

• The dataset construction and cleaning rules used to reduce mock runs, missing data, and interrupted workflows.
• The full confusion matrix and per-category performance results for each evaluated model.
• The customer-environment tool stacks and workflow categories that shaped the triage decisions.
• The benchmark's reasoning tags and annotation method for separating analyst disagreement from model error.

👉 The full Legion AI article includes the dataset methodology, per-use-case results, and confusion matrix details.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to connect identity controls to broader security operations.