Sigma rules and MITRE ATT&CK labels still challenge LLMs

At a glance

What this is: This benchmark tests how well LLMs map Sigma detections to MITRE ATT&CK techniques, and the best models still fall short of complete technique coverage.

Why it matters: For SOC, detection engineering, and SIEM teams, the gap matters because technique mapping shapes rule enrichment, triage quality, and how reliably security operations can turn detections into adversary context.

By the numbers:

• Claude Opus 4.5 achieved the highest hierarchical F1 score at 66% in a benchmark of 2,733 Sigma detection rules.
• Gemini 3.0 Flash achieved the highest recall at 71%, making it the strongest coverage model in the test.
• DeepSeek v3.2 led the open-weight models at 48% F1, but it remained well below the top frontier systems.
• The benchmark evaluated 12 models against 2,733 Sigma detection rules from SigmaHQ.

👉 Read Cotool's benchmark analysis of LLM performance on Sigma-to-ATT&CK mapping

Context

MITRE ATT&CK mapping is a classification problem with operational consequences. When detection logic is tagged poorly, analysts lose consistency in triage, reporting, and threat hunting, and downstream automation inherits that ambiguity. This benchmark looks at how well language models can infer ATT&CK techniques from Sigma rules, which is relevant to SOC enrichment and detection engineering.

The identity connection here is indirect but real. ATT&CK enrichment often feeds privilege abuse investigations, credential access analysis, and NHI-related detections inside SIEM and SOAR workflows. The current starting position is typical of the market: models can assist with classification, but they do not remove the need for human validation or control-aware review.

Sigma rules are structured detections written by security teams, while MITRE ATT&CK is a behavioural taxonomy for adversary tradecraft. The benchmark strips MITRE tags from the input, forcing models to infer techniques from the rule itself rather than from labels. That makes it a useful test of whether a model understands the detection pattern or is merely matching metadata.

Key questions

Q: How should security teams use LLMs to map Sigma rules to MITRE ATT&CK?

A: Use them as enrichment assistants, not as final authorities. Let the model propose likely technique IDs, then have analysts validate the result against the rule logic, logging context, and local detection conventions. This works best when the output feeds review queues or content pipelines, not auto-published detections.

Q: Why do ATT&CK labels from Sigma rules often need human review?

A: Because Sigma expresses detection logic, while ATT&CK expresses adversary behaviour. A rule can legitimately map to more than one technique, or to a parent technique when the specific sub-technique is ambiguous. Human review resolves that ambiguity and prevents false confidence from entering reporting or automation.

Q: What breaks when ATT&CK technique mapping is inconsistent across detections?

A: Inconsistent mapping weakens hunting, distorts reporting, and makes it harder to compare detections across teams or environments. It also reduces the value of SIEM enrichment and can hide patterns tied to credential access, privilege escalation, or lateral movement. Standardisation matters because labels are part of the control fabric.

Q: How do teams know if ATT&CK enrichment is actually helping detection engineering?

A: Measure analyst correction rates, coverage of important technique families, and whether enriched rules improve triage speed or hunt precision. If the model adds noise without improving review quality, it is not helping. The useful signal is not perfect accuracy, but whether it makes detection content more consistent and more actionable.

Technical breakdown

Why Sigma-to-ATT&CK classification is hard

Sigma rules compress a detection idea into a small set of indicators, but ATT&CK techniques often sit one abstraction level higher. A rule may detect PowerShell, encoded command lines, or process creation, yet the right label could be a parent technique or a more specific sub-technique depending on context. The benchmark’s hierarchical scoring reflects that ambiguity by giving partial credit when a model gets the parent technique right. That matters because detection engineering is not just about naming malware or commands, but about expressing adversary behaviour consistently across environments.

Practical implication: teams should treat AI-generated ATT&CK labels as enrichment candidates, not authoritative truth.

What hierarchical F1 reveals about model usefulness

Hierarchical F1 rewards semantic proximity, so a model can score reasonably well even when it misses exact sub-techniques. That is useful in security operations because many workflows need approximate mapping first, then analyst confirmation. But the score also hides a key issue: incomplete ground truth and parent-child overlap can make a model look better than it is for production use. In other words, a tool can be directionally useful for classification while still being unreliable for automated rule publishing, reporting, or detection content normalization.

Practical implication: validate model output against your own rule corpus before using it in pipelines.

Why recall is the more operationally relevant signal

The benchmark notes that community-contributed Sigma tags are often incomplete, which makes recall a better indicator than precision. In detection engineering, missing a relevant technique can be more harmful than adding an extra label, because under-tagging reduces visibility into adversary behaviour and weakens hunt logic. That creates a governance problem as much as a technical one: model outputs need a review path that accepts imperfect suggestions while preventing false certainty from entering SIEM, SOAR, or threat reporting workflows.

Practical implication: optimise for analyst review coverage first, then tighten precision through corpus-specific tuning.

NHI Mgmt Group analysis

Technique mapping is becoming a governance layer, not just a classification task. When teams use LLMs to infer ATT&CK techniques from Sigma rules, they are not merely automating annotation. They are influencing how detections are prioritised, how hunts are scoped, and how control gaps are reported to leadership. That makes technique mapping part of operational governance, especially when SIEM and SOAR workflows depend on consistent labels. The practitioner conclusion is simple: treat ATT&CK enrichment as controlled metadata, not free-form AI output.

The benchmark exposes a detection engineering translation gap. Sigma describes what a rule sees, while ATT&CK describes what an adversary is doing. Models struggle because those layers do not always align cleanly, especially when a rule could map to a parent technique, a sub-technique, or multiple behaviours. This is a useful reminder that the security value is in translation quality, not raw model fluency. Practitioners should expect human review to remain part of the pipeline.

Technique inference is relevant to NHI and privilege abuse investigations. ATT&CK labels often feed investigations involving credential access, service accounts, and privilege escalation, which is where identity teams and SOC teams intersect. If a model mislabels those events, it can distort the picture of how non-human identities are being abused. The result is weaker linkage between detection engineering and identity governance, so teams should preserve analyst oversight wherever identity-sensitive detections are involved.

Coverage matters more than exactness in operational enrichment. Gemini 3.0 Flash’s higher recall suggests that broader coverage can be preferable when the goal is surfacing likely techniques for analyst triage. But coverage without control creates noise, and noise is expensive in security operations. The named concept here is ATT&CK translation debt: the gap between a rule’s raw detection logic and the behavioural label needed for reporting, hunting, and response. The practitioner conclusion is to reduce that debt through reviewable, corpus-specific workflows.

Benchmark scores should not be mistaken for deployment readiness. A model that performs well on Sigma-to-ATT&CK classification may still fail when the rule corpus, logging standards, or ATT&CK usage conventions change. That is especially true in environments where detections are tied to compliance narratives or identity investigations. The practical takeaway is to benchmark on your own detections, measure analyst correction rates, and keep the final authority with humans.

What this signals

Detection engineering teams should expect AI-assisted ATT&CK mapping to become a normal part of content operations, but only when the output is governed like any other security metadata. The practical signal is not whether models can guess the right technique sometimes, but whether they can reduce analyst toil without changing the control owner for the rule.

ATT&CK translation debt: the gap between raw Sigma logic and the behavioural label needed for hunting and reporting will remain a recurring source of operational friction. Teams that use MITRE ATT&CK Enterprise Matrix and the NIST SP 800-53 Rev 5 Security and Privacy Controls together will be better positioned to keep enrichment consistent with control intent.

For practitioners

• Use AI for first-pass ATT&CK enrichment only Route model output into analyst review queues rather than auto-publishing labels into SIEM content, because hierarchical matches can still hide technique-level mistakes.
• Measure correction rates on your own Sigma corpus Track how often analysts change predicted technique IDs across your highest-value detections, then use that error pattern to decide whether the model is fit for enrichment.
• Separate parent-technique and sub-technique workflows Apply different review thresholds when the model predicts a parent ATT&CK technique versus a sub-technique, since the operational risk of over- or under-specific mapping is not the same.
• Prioritise identity-sensitive detections for manual validation Require human confirmation for rules linked to credential access, service account abuse, and privilege escalation so NHI-related investigations do not inherit weak labels.

Key takeaways

• LLMs can help map Sigma rules to ATT&CK, but the benchmark shows they still need analyst oversight to stay operationally reliable.
• Recall is the more useful signal for detection enrichment, yet higher coverage does not eliminate the governance risk of mislabelled techniques.
• Security teams should treat ATT&CK enrichment as controlled metadata, especially where detections intersect with credential access and identity abuse.

Key terms

• Sigma Rule: A Sigma rule is a platform-neutral detection pattern written to describe suspicious activity in logs or telemetry. It standardises detection logic so teams can translate one rule into multiple SIEM or hunting environments without rewriting the underlying behavioural idea.
• MITRE ATT&CK Technique: An ATT&CK technique is a named adversary behaviour used to describe how attackers operate, such as credential access, lateral movement, or exfiltration. It gives analysts a common language for mapping detections to threat behaviour and for comparing coverage across environments.
• Hierarchical F1 Score: Hierarchical F1 score is an evaluation method that gives partial credit when a prediction lands on a parent or closely related category rather than the exact label. In security classification tasks, it helps measure semantic usefulness when taxonomies have nested relationships.
• Detection Engineering: The discipline of designing, testing, and maintaining detection logic so it remains useful against real attacker behaviour. It covers telemetry selection, rule quality, false-positive management, and the operational workflow needed to keep alerts actionable.

What's in the full report

Cotool's full analysis covers the benchmark methodology and model-by-model results this post intentionally leaves at a higher level:

• Detailed F1, precision, recall, cost, and latency results for all 12 evaluated models
• The benchmark scoring method, including hierarchical credit for parent and sub-technique predictions
• Sample Sigma rule inputs showing how labels were inferred from detection logic
• Model recommendation notes explaining when higher recall may be more useful than exact precision

👉 Cotool's full post includes the benchmark methodology, model comparisons, and score breakdowns across all 12 systems.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is a strong fit for practitioners building identity-aware controls across modern security programmes.