Topic starter
06/09/2026 9:34 am
TL;DR: A benchmark of 163 real-world security triage decisions found Gemini 3 Pro performed best overall at 74.8%, while the strongest model changed by use case, with different winners for phishing, account takeover, and network investigations, according to Legion AI. The result is a reminder that security teams need formal evaluation against their own workflows, not a one-model strategy.
Editorial analysis by NHI Mgmt Group, based on content published by Legion AI: “Benchmarking Large Language Models for Automated Security Triage”.
By the numbers:
- Legion AI evaluated 163 real-world security triage decisions after cleaning a 196-case dataset.
Key questions
Q: How should security teams evaluate LLMs for triage decisions?
A: Evaluate LLMs against the exact workflows, evidence types, and decision labels they will see in production.
Q: Why does the best LLM for SOC triage change by use case?
A: Because triage quality depends on the evidence pattern, the decision options available, and the kind of judgment required at the workflow node.
Q: What are the signs that an AI-assisted triage workflow is failing to reduce analyst workload?
A: The clearest signs are persistent manual research on familiar alerts, inconsistent dispositions for repeat patterns, and little movement in queue volume or decision time.
Practitioner guidance
- Define separate benchmark sets for each triage queue Build distinct evaluation slices for phishing, account takeover, network, and any identity-heavy queues so scores reflect the actual decision path each workflow uses.
- Test the model against your own decision options Use the exact response choices, escalation thresholds, and workflow summaries your analysts see in production, rather than generic prompt tests.
- Audit upstream workflow quality before comparing models Check for missing steps, ambiguous labels, and inconsistent analyst annotations, because workflow defects can look like model failure.
Bottom line: Security triage models do not generalise evenly across investigation types, so a single benchmark score can hide queue-specific weaknesses.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
06/09/2026 9:34 am
Model choice is now a workflow governance problem, not a benchmark vanity metric. The key finding is not just that one model scored highest overall, but that the best performer changed by use case. That means the control question for security leaders is whether a model is sufficiently accurate for a specific triage queue, not whether it wins a broad leaderboard. Practitioners should treat evaluation as part of operational governance.
A few things that frame the scale:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: Should teams use one model across phishing, account takeover and network triage?
A: Only if it clears your own acceptance threshold in each queue. The safer approach is to compare models per use case, because the best overall model may not be the best for every investigation type, and SOC performance depends on the weakest supported workflow.
👉 Read our full editorial: Benchmarking LLMs for security triage shows model choice is use-case specific