Topic starter
18/08/2026 2:40 pm
TL;DR: Among 2,733 Sigma detection rules, Claude Opus 4.5 reached the highest hierarchical F1 score at 66%, while Gemini 3.0 Flash achieved the best recall at 71% and open-weight DeepSeek v3.2 led its class at 48% F1, according to Cotool. The result is a reminder that detection engineering workloads still depend on accurate adversary-technique mapping, not just model fluency.
Editorial analysis by NHI Mgmt Group, based on content published by Cotool: “Sigma Detection Classification”.
By the numbers:
- The benchmark evaluated 2,733 Sigma detection rules across 12 models.
- DeepSeek v3.2 led open-weight models at 48% F1.
Key questions
Q: What should teams do when Sigma rule labels and ATT&CK mappings are incomplete?
A: Treat the labels as a draft classification layer, not a source of truth.
Q: Why does ATT&CK mapping fail even when an LLM understands the Sigma rule text?
A: Because the task requires security reasoning, not just language understanding.
Q: How do you know if a model is actually good at detection engineering classification?
A: Check whether it predicts techniques that match the detection intent at the granularity your programme uses.
Practitioner guidance
- Define the required ATT&CK granularity Set whether your programme needs parent techniques, sub-techniques, or both before you use LLMs to tag Sigma rules.
- Use recall-first evaluation for noisy labels Weight false negatives more heavily than false positives when your Sigma ground truth is community contributed and likely incomplete.
- Validate model output against rule intent Compare each predicted technique to the detection logic, logsource, and description, not just to the existing tag set.
Bottom line: The benchmark shows that LLM performance on Sigma-to-ATT&CK mapping is uneven, with the best model still far from perfect on a task that underpins detection metadata quality.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
18/08/2026 3:24 pm
Technique mapping is a governance control, not a labeling convenience. ATT&CK tags are used to describe detection coverage, measure control breadth, and prioritise hunting work, so weak mapping quality becomes a reporting problem as well as an analysis problem. When a model can produce a plausible label without understanding the detection logic, the organisation risks building dashboards on unstable metadata. Practitioners should treat ATT&CK enrichment as part of detection governance, not a cosmetic layer.
A question worth separating out:
Q: Should teams prioritise recall or precision when using LLMs for ATT&CK enrichment?
A: Prioritise recall when the cost of missing a technique is higher than the cost of over-tagging, which is common in detection engineering. Precision still matters, but community-contributed labels are often incomplete, so some apparent false positives may be valid techniques the rule author did not tag. Balance the metric to the use case.
👉 Read our full editorial: Sigma rules and MITRE ATT&CK labels still challenge LLMs