Sigma rules and MITRE ATT&CK labels still challenge LLMs

At a glance

What this is: This benchmark tests how well LLMs map Sigma detection rules to MITRE ATT&CK techniques, with Claude Opus 4.5 leading overall F1 and Gemini 3.0 Flash leading recall.

Why it matters: It matters because detection engineering teams using LLMs for enrichment or classification need to understand where model recall, precision, and hierarchy-aware scoring can distort ATT&CK coverage.

By the numbers:

• The benchmark evaluated 2,733 Sigma detection rules across 12 models.
• Claude Opus 4.5 achieved the highest hierarchical F1 score at 66%.
• Gemini 3.0 Flash achieved the highest recall at 71%.
• DeepSeek v3.2 led open-weight models at 48% F1.

Context

Sigma rule to ATT&CK mapping is a classification problem, but in practice it sits inside detection engineering governance. Teams use ATT&CK labels to describe coverage, prioritise detections, and communicate adversary technique focus, so errors in technique mapping can distort both operational reporting and threat modelling. This benchmark matters because the quality of that mapping depends on the model's understanding of adversary tradecraft, not just its general language ability.

The article also exposes a familiar gap in security automation: community-authored detection content is not a perfect ground truth, so evaluation has to tolerate incomplete labels and hierarchy-aware ambiguity. That makes this a useful signal for teams considering LLM-assisted detection triage, enrichment, or ATT&CK tagging. The core issue is not whether a model can produce a plausible label, but whether it can support consistent, defensible detection metadata at scale.

Key questions

Q: What should teams do when Sigma rule labels and ATT&CK mappings are incomplete?

A: Treat the labels as a draft classification layer, not a source of truth. Validate model output against the Sigma rule logic, then use analyst review for any technique labels that will drive reporting, hunting, or prioritisation. Incomplete labels distort precision, so recall and coverage of intended behaviour are usually the more useful measures.

Q: Why does ATT&CK mapping fail even when an LLM understands the Sigma rule text?

A: Because the task requires security reasoning, not just language understanding. The model has to infer adversary behaviour from detection logic, logsource context, and short descriptions, then map that behaviour to the right technique family or sub-technique. Fluent output can still miss the operational meaning of the rule.

Q: How do you know if a model is actually good at detection engineering classification?

A: Check whether it predicts techniques that match the detection intent at the granularity your programme uses. If ground truth is noisy, judge the model by coverage of intended behaviours, not only by strict tag agreement. A useful model should improve tagging consistency without obscuring where human review is still needed.

Q: Should teams prioritise recall or precision when using LLMs for ATT&CK enrichment?

A: Prioritise recall when the cost of missing a technique is higher than the cost of over-tagging, which is common in detection engineering. Precision still matters, but community-contributed labels are often incomplete, so some apparent false positives may be valid techniques the rule author did not tag. Balance the metric to the use case.

Technical breakdown

How hierarchical ATT&CK scoring changes Sigma evaluation

Hierarchical scoring gives partial credit when a model predicts a parent technique instead of the exact sub-technique. That matters because Sigma rules often describe behaviour that maps to a technique family rather than one precise sub-technique, and ATT&CK itself is hierarchical by design. In this benchmark, that structure makes F1 less like a simple label-match score and more like a measure of semantic alignment with adversary tradecraft. For practitioners, the useful question is not only whether the model is right, but whether it is right at the level your detection taxonomy needs.

Practical implication: decide whether your pipeline needs exact sub-technique labels or only defensible parent-technique coverage before using LLM outputs.

Why recall matters more than precision in detection tagging

The article notes that community-contributed ground truth labels are often incomplete, which means a model can appear to underperform when it actually predicts a valid but untagged technique. In that setting, recall is a better indicator of whether the model covers the intended adversary behaviour, while precision can be artificially punished by missing human annotations. For detection engineering, this is a governance issue as much as a machine-learning one: the evaluation target must reflect how the organisation uses ATT&CK labels, not just how the benchmark is scored.

Practical implication: tune evaluation to your label quality and treat low precision cautiously when the source taxonomy is incomplete.

Why Sigma rules are still hard for LLMs to classify

Sigma rules compress detection logic, logsource context, and short descriptive text into a compact artifact that still requires adversary-context reasoning. The model has to infer intent from clues such as command-line flags, script behaviour, or suspicious tool usage, then map that behaviour to one or more ATT&CK techniques. That is a harder task than ordinary text classification because the signal is operational, not rhetorical. For security teams, the result is a reminder that detection metadata quality depends on technique-level reasoning, not generic model fluency.

Practical implication: use LLMs as enrichment assistants for Sigma, not as unsupervised authors of ATT&CK mappings.

NHI Mgmt Group analysis

Technique mapping is a governance control, not a labeling convenience. ATT&CK tags are used to describe detection coverage, measure control breadth, and prioritise hunting work, so weak mapping quality becomes a reporting problem as well as an analysis problem. When a model can produce a plausible label without understanding the detection logic, the organisation risks building dashboards on unstable metadata. Practitioners should treat ATT&CK enrichment as part of detection governance, not a cosmetic layer.

Recall is the more meaningful signal when human labels are incomplete. The article correctly points to community-contributed Sigma labels as an imperfect ground truth, which means some apparent false positives may actually be missed annotations. That shifts the evaluation question from simple accuracy to coverage of intended tradecraft. Teams adopting LLM-assisted classification should calibrate scoring to their own annotation quality before making model-selection decisions.

Detection engineering exposes the boundary between language competence and security reasoning. A model can parse Sigma syntax and still miss the adversary technique it implies, especially where the rule encodes behaviour rather than named malware. That is why benchmark results for detection tagging do not transfer cleanly from general benchmark performance. Security teams should expect better results from models that can reason over operational clues, not just generate fluent outputs.

Hierarchical ATT&CK awareness should become part of the evaluation contract. Parent technique and sub-technique confusion is not a minor scoring detail, because many detection programmes only need stable family-level attribution while others need finer precision. The article shows that hierarchy-aware scoring better reflects the practical reality of detection metadata. Practitioners should define the required ATT&CK granularity before they measure model quality.

MITRE ATT&CK taxonomy alignment belongs in the detection pipeline, not after it. If Sigma rules are being classified by LLMs, the model output should be validated against rule intent and threat-model assumptions before it is used in reporting or triage. The right control is not just post-hoc review, but a structured quality gate for adversary-technique mapping. Practitioners should integrate that check into detection engineering workflows.

What this signals

ATT&CK label quality will become a governance issue as more teams automate detection enrichment. Once LLMs start producing technique metadata at scale, the control question shifts from output generation to validation and reviewer accountability. The programme risk is not just a wrong label, but a wrong label entering dashboards, metrics, and executive reporting as if it were authoritative.

Hierarchical scoring should be built into any internal evaluation of Sigma classification. Parent and sub-technique distinctions matter because different teams consume ATT&CK labels at different levels of precision. A model that is acceptable for broad coverage may still be unsuitable for engineering workflows that require exact sub-technique fidelity.

Technique mapping is the kind of task where model fluency can mask poor security reasoning. Detection engineering depends on understanding what the rule is trying to catch, not just translating words into labels. Practitioners should therefore test LLMs against real Sigma content before using them for enrichment in production pipelines.

For practitioners

• Define the required ATT&CK granularity Set whether your programme needs parent techniques, sub-techniques, or both before you use LLMs to tag Sigma rules.
• Use recall-first evaluation for noisy labels Weight false negatives more heavily than false positives when your Sigma ground truth is community contributed and likely incomplete.
• Validate model output against rule intent Compare each predicted technique to the detection logic, logsource, and description, not just to the existing tag set.
• Insert human review before ATT&CK reporting Require analyst approval for technique labels that will feed dashboards, hunting plans, or board reporting.

Key takeaways

• The benchmark shows that LLM performance on Sigma-to-ATT&CK mapping is uneven, with the best model still far from perfect on a task that underpins detection metadata quality.
• Recall is the more useful signal than precision when the ground truth is community contributed and may omit valid technique labels.
• Teams should treat ATT&CK enrichment as a governed detection workflow, with human review and hierarchy-aware evaluation before model output is trusted.

Key terms

• Sigma Rule: A Sigma rule is a platform-neutral detection pattern written to describe suspicious activity in logs or telemetry. It standardises detection logic so teams can translate one rule into multiple SIEM or hunting environments without rewriting the underlying behavioural idea.
• MITRE ATT&CK: MITRE ATT&CK is a structured knowledge base of adversary tactics and techniques. Security teams use it to describe, compare, and prioritise detection coverage, hunting hypotheses, and threat intelligence in a consistent way.
• Hierarchical F1 Score: Hierarchical F1 score is an evaluation method that gives partial credit when a prediction lands on a parent or closely related category rather than the exact label. In security classification tasks, it helps measure semantic usefulness when taxonomies have nested relationships.
• Detection Engineering: The discipline of designing, testing, and maintaining detection logic so it remains useful against real attacker behaviour. It covers telemetry selection, rule quality, false-positive management, and the operational workflow needed to keep alerts actionable.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and identity lifecycle controls. It helps practitioners connect identity discipline to the broader security programmes they run every day.