Benchmarking LLMs for security triage shows model choice is use-case specific

At a glance

What this is: Legion AI's benchmark shows that LLM triage performance is use-case specific, with different models winning across phishing, account takeover and network investigations.

Why it matters: Security teams using LLMs for triage need to test against their own decision paths, because the model that works best in one investigation type may be weaker in another.

By the numbers:

• Legion AI evaluated 163 real-world security triage decisions after cleaning a 196-case dataset.

Context

Large language models can support security triage, but their value depends on the decision context they are asked to handle. A model that performs well on one investigation type may underperform on another when the workflow, evidence mix, and escalation threshold change. For IAM and NHI programmes, that matters because triage quality shapes whether suspicious access, compromised credentials, or benign activity is handled correctly.

This benchmark looks at security operations as a governed decision system, not just a classification task. The article's core message is that triage models must be evaluated against the organisation's own workflows, because the same reasoning pattern does not generalise cleanly across phishing, account takeover and network investigations. That is typical of operational AI in security, where context often determines reliability more than raw benchmark score.

Key questions

Q: How should security teams evaluate LLMs for triage decisions?

A: Evaluate LLMs against the exact workflows, evidence types, and decision labels they will see in production. Use separate test sets for phishing, account takeover, and network cases, then measure not just accuracy but disagreement patterns, escalation quality, and how often missing context changes the outcome.

Q: Why does the best LLM for SOC triage change by use case?

A: Because triage quality depends on the evidence pattern, the decision options available, and the kind of judgment required at the workflow node. Phishing, account takeover, and network investigations stress different reasoning paths, so a model can be strong in one queue and only average in another.

Q: What are the signs that an AI-assisted triage workflow is failing to reduce analyst workload?

A: The clearest signs are persistent manual research on familiar alerts, inconsistent dispositions for repeat patterns, and little movement in queue volume or decision time. If analysts still have to dig through closed tickets to understand obvious lookalikes, the workflow is not delivering enough context. A useful system should quickly identify close matches and separate repeat noise from alerts that need escalation.

Q: Should teams use one model across phishing, account takeover and network triage?

A: Only if it clears your own acceptance threshold in each queue. The safer approach is to compare models per use case, because the best overall model may not be the best for every investigation type, and SOC performance depends on the weakest supported workflow.

Technical breakdown

Why triage benchmarks change by use case

Security triage is not a single problem. Phishing, account takeover, and network investigation all require different evidence weighting, different false-positive tolerance, and different escalation paths. A model can be strong at one pattern because the signal is compact and familiar, then weaken on another where context is partial or the decision depends on prior workflow steps. That is why aggregate scores can hide the operational risk that matters to a SOC: the model may be right often enough overall while still being unreliable in a specific queue.

Practical implication: Measure model performance separately for each triage queue before allowing it into production workflows.

What a triage decision node actually tests

A triage decision node is the point in a workflow where the model must choose among predefined response options using the evidence collected so far. That makes the test closer to operational judgment than generic text classification. It also means the quality of the upstream workflow matters, because missing evidence, poor context assembly, or inconsistent decision labels can distort the model's apparent accuracy. In security operations, the triage system is part model and part process, so benchmark design has to reflect both.

Practical implication: Validate the workflow inputs and decision options as carefully as the model output itself.

Why federated evaluation matters for security data

Security teams rarely evaluate triage with a single clean data source. This study used summary statistics from multiple customer environments without mixing underlying tenant data, which is a useful pattern for preserving isolation while still comparing performance. That approach is especially relevant when the triage subject may touch identity events, account misuse, or machine credentials, because cross-tenant data handling can create governance and privacy problems of its own. The evaluation method is part of the control surface, not just the research design.

Practical implication: Use isolated, workflow-specific evaluation datasets when assessing AI for SOC or identity triage.

Threat narrative

Attacker objective: The practical objective is to get malicious activity misclassified or deprioritised so that response is delayed or misrouted.

1. Entry begins when a user, account, or network signal enters a security workflow and the model must interpret the evidence available at that point.
2. Escalation occurs when the model's decision changes whether the case is closed, reviewed, or investigated further, making classification errors operationally material.
3. Impact follows when the wrong triage choice delays response to a real phishing, account takeover, or network incident, or creates avoidable analyst workload.

Breaches seen in the wild

• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Model choice is now a workflow governance problem, not a benchmark vanity metric. The key finding is not just that one model scored highest overall, but that the best performer changed by use case. That means the control question for security leaders is whether a model is sufficiently accurate for a specific triage queue, not whether it wins a broad leaderboard. Practitioners should treat evaluation as part of operational governance.

Triage accuracy depends on evidence shape, which is why generic AI assessments miss the real risk. Security investigations are structured around partial context, workflow steps, and predefined response options. A model that looks strong in a static prompt test may fail when it has to reason over multiple tool outputs and organisation-specific knowledge. That makes decision-node evaluation the right lens for SOC and identity use cases.

Use-case specificity is a named risk in security AI: triage generalisation gap. This article shows that a single model strategy can hide unstable performance across phishing, account takeover and network workstreams. The gap is operationally important because it affects both alert quality and analyst trust. Security teams should assume that model capability is local to the workflow, not universal across the SOC.

Identity-rich investigations raise the stakes for model governance. Account takeover and network triage often depend on credential, session, and access-path interpretation, which is exactly where weak reasoning can create false reassurance. When AI is used near IAM or NHI telemetry, poor triage can let suspicious access patterns age into incidents. The practitioner conclusion is to align AI evaluation with identity-sensitive decision paths.

Federated evaluation is the right pattern for cross-tenant security AI testing. The article's method shows that teams can compare model behaviour without collapsing customer data into a shared pool. That matters for regulated environments and for any SOC handling identity events or sensitive security telemetry. The governance lesson is to test locally, compare consistently, and avoid treating training-style benchmarks as operational proof.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Security Platform Buyer's Guide

What this signals

Triage AI needs queue-level governance, not just model selection. Security teams should expect different performance curves across phishing, account takeover, and network investigations, so the evaluation unit has to be the workflow queue rather than the model family. That is especially true where identity signals are part of the decision path, because access context changes the cost of a wrong call.

Triaging identity events with LLMs creates a governance boundary at decision time. When a model is asked to interpret account misuse or suspicious access, the risk is not only classification accuracy but also whether the model understands the organisation's process, evidence, and escalation rules. The operating rule should be simple: if the workflow changes, re-test the model.

For practitioners

• Define separate benchmark sets for each triage queue Build distinct evaluation slices for phishing, account takeover, network, and any identity-heavy queues so scores reflect the actual decision path each workflow uses.
• Test the model against your own decision options Use the exact response choices, escalation thresholds, and workflow summaries your analysts see in production, rather than generic prompt tests.
• Audit upstream workflow quality before comparing models Check for missing steps, ambiguous labels, and inconsistent analyst annotations, because workflow defects can look like model failure.
• Set a minimum accuracy bar per use case Require model-specific thresholds for each queue, then block production use in any workflow where the model falls below the agreed operating floor.

Key takeaways

• Security triage models do not generalise evenly across investigation types, so a single benchmark score can hide queue-specific weaknesses.
• The practical unit of evaluation is the workflow decision node, because that is where evidence quality, escalation thresholds and model reasoning intersect.
• Teams should set acceptance thresholds per use case and keep testing when workflows, tools or response options change.

Key terms

• Triage Decision Node: A triage decision node is a fixed point in an investigation workflow where a model or analyst chooses from predefined outcomes after reviewing gathered evidence. It is important because it turns unstructured security work into a controlled, measurable decision process that can be benchmarked across cases and teams.
• Use-Case Specific Evaluation: Use-case specific evaluation measures a model against the exact task, workflow, and decision context where it will be used. In security, this matters because a general benchmark can hide weak performance in one queue even when overall scores look acceptable.
• Federated Evaluation: Federated evaluation compares model behaviour across separate environments without pooling the underlying tenant data into one shared dataset. It reduces data exposure and preserves organisational boundaries while still allowing consistent benchmarking across multiple customers or business units.
• Workflow context: Workflow context is the supporting evidence that shows why an identity action happened, such as a help-desk ticket, verification step, or approval trail. Without it, detection systems see only the event and often misclassify legitimate administrative work as suspicious.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course, the industry's only accredited NHI security programme. It is designed for practitioners who need to connect identity controls, machine access, and AI-driven operations.