LLMs for Detection Engineering Guide: Writing, Testing and Tuning Rules Safely

Large language models can draft detection rules, translate queries and summarise threat reports in seconds, but they also invent field names, map behaviours to the wrong ATT&CK technique and write logic that looks right and never fires. NIST calls this confabulation: generative systems confidently presenting false content. In detection engineering a confabulated rule is worse than no rule, because it creates the belief that a technique is covered. This guide explains where LLMs genuinely help detection teams, the specific ways they fail, and the guardrails (grounding, validation, testing and review) that turn LLM output into production-grade detections. It is for detection engineers, SOC leads and security engineering managers. The core discipline is covered in the Detection Engineering Guide; this page covers using LLMs inside it.

Key takeaways

• Use LLMs for drafts, never for verdicts. Let the model draft rules, queries, documentation and test ideas; let validators, test data and a human reviewer decide what ships.
• Ground the model in your schema. Most LLM rule failures are hallucinated or mismatched fields and log sources. Give it your actual tables, field names and sample events, and reject anything that references fields outside them.
• Every generated rule needs a test. A rule is not done until it fires on a true positive (for example an Atomic Red Team test or a recorded sample) and stays quiet on a benign baseline.
• Treat logs and reports as untrusted input. When an LLM reads attacker-controlled data such as command lines, email bodies or threat reports, that content can carry prompt injection. Keep the model read-only and away from response actions it could be steered into.
• Measure the assistant, not just the rules. Track how many generated rules pass validation first time, how many need logic changes in review, and false positive rates after deployment.

Where LLMs help and where they do not

How LLM-written detections fail

OWASP lists misinformation (LLM09:2025) as a core LLM risk, driven by hallucination and overreliance. In detection work it shows up in predictable forms.

• Hallucinated fields and tables: the rule references a field that does not exist in your data model, or exists under a different name in your SIEM. The query runs, returns nothing and looks clean.
• Wrong log source: a process creation rule written against a field set from a different product, or a Windows Security event rule written for data you only collect through Sysmon.
• Logic inversion and operator errors: and where or was needed, a misplaced not, or a condition that requires two mutually exclusive values on one event.
• Over-broad wildcards: rules that match every PowerShell launch rather than the suspicious pattern, producing alert floods that are then muted.
• String matching that misses variants: exact matches on one tool name or command spelling that an attacker changes trivially.
• Invented or revoked ATT&CK IDs: technique IDs that never existed, or that MITRE has revoked or restructured. ATT&CK v20, for example, revoked several older defence evasion IDs, so a model trained on older data will suggest them.
• Confident false explanations: a convincing description of what a rule does that does not match the query, which misleads reviewers who read the prose rather than the logic.

None of these failures produce an error. They produce silence, which is why each guardrail below exists.

Guardrails: a pipeline for LLM-assisted rules

Put the LLM inside the detection-as-code pipeline you already use, not beside it.

1. Ground the prompt: supply the target language, the exact table and field list for the log source, two or three real sample events (sanitised), and the ATT&CK technique to cover. Ask for Sigma where possible, because it is vendor-neutral and converts deterministically.
2. Require structure: for Sigma, require title, id (UUID version 4), status: experimental, logsource, detection, falsepositives, level and tags, as defined in the Sigma Rules Specification (version 2.1.0). LLM drafts always enter as experimental, never stable.
3. Validate syntax and schema automatically: parse the rule (pySigma for Sigma, the Elastic detection-rules validate-all and KQL parser for Elastic, a query parser or dry run for KQL and SPL), then check every referenced field against the data model and reject unknown fields.
4. Validate mappings: check every ATT&CK tag against the current ATT&CK release and reject revoked or deprecated IDs.
5. Convert deterministically: use sigma convert with the correct backend and processing pipeline (for example a Sysmon or ECS pipeline) rather than asking the model to write each SIEM dialect.
6. Test for true positives: run the matching Atomic Red Team test, a recorded attack sample or a replay in a lab, and confirm the rule fires.
7. Test for noise: run the rule against a representative benign window (for example 7 to 30 days of production data) and record the hit count.
8. Human review: a detection engineer reviews the logic, not the explanation, and signs off. Review effort should be the same as for a human-written rule.
9. Deploy in stages: start in a monitoring or low-severity state, promote to test and then stable after a defined period with acceptable precision. Roll back by reverting the commit, which the pipeline redeploys.

Rules flowing through this pipeline are tested with the same rigour whoever wrote them. See the Sigma Rules Guide for rule structure and conversion.

Worked example (illustrative): from threat report to tested rule

This composite example shows how the guardrails catch typical defects; it is illustrative, not a specific incident. A threat report describes an intrusion in which the actor dumped credentials from LSASS memory (ATT&CK T1003.001 LSASS Memory) using a signed system binary. The engineer gives the model the relevant paragraph, the process access and process creation field lists from the organisation's Sysmon data, and two sanitised sample events, and asks for a Sigma rule.

• What the draft got right: a sensible log source (process access), the target image, and a list of known false positives such as endpoint security agents.
• What validation caught: one field name came from a different product's schema and was rejected by the field check; one tag referenced a technique ID that ATT&CK no longer uses and was rejected by the ID check.
• What testing caught: the access mask condition was an exact match on one value, so the Atomic Red Team test for the same technique, which uses a different value, did not fire. The engineer changed the condition to cover the documented variants and the test passed.
• What review caught: the model's proposed exclusion for a backup agent used a directory wildcard an attacker could write to; it was replaced with the agent's signed publisher.
• Outcome: the rule entered as experimental, ran for two weeks with an acceptable alert volume and was promoted to test. Total engineer time was lower than writing from scratch, but only because the validators and the test caught three defects that review alone might have missed.

Testing and tuning with LLMs

Generating test cases

• Variant lists: ask the model for ways an attacker could vary the behaviour (renamed binaries, alternate parent processes, encoded arguments) and turn each into a test case. This exposes brittle string matches.
• Benign look-alikes: ask for legitimate administrative activity that resembles the attack, then check the rule against real examples of that activity.
• Curated execution only: execute tests from maintained libraries such as Atomic Red Team in an isolated lab. Do not run model-generated commands on production systems.

Tuning false positives

• Summarise, then decide: an LLM can cluster a week of false positives by parent process, user or host and propose exclusions. The engineer decides which are safe.
• Test exclusions against attacks: re-run the true positive tests after adding an exclusion; a common failure is excluding a path or account an attacker can also use.
• Prefer narrow exclusions: exact hashes, signed publishers or specific service accounts, never broad directory wildcards.

Securing the LLM in the detection workflow

The model and its integrations become part of the SOC's attack surface. Treat the assistant as a non-human identity with access to sensitive data.

Vendor assistants follow the same pattern. Microsoft's documentation for the Security Copilot query assistant in Defender advanced hunting, for example, states that it turns natural language into KQL, runs read-only queries within the user's existing permissions, and that users should review the generated query and its results before acting on them.

Choosing an approach

Small organisations: use LLMs for explaining and documenting rules from maintained repositories (SigmaHQ, vendor rule sets) rather than writing new ones, until you can test. Large organisations: integrate generation into CI with automated schema and ATT&CK validation, and track assistant metrics per team.

Measuring whether it works

• First-pass validation rate: share of generated rules that pass syntax, schema and ATT&CK validation without edits.
• Review change rate: share of generated rules whose detection logic changed during human review (a high rate means the assistant is not saving time).
• True positive test pass rate: share of new rules that fire on their test before deployment (target 100%).
• Precision after 30 days: true positives divided by all alerts for rules promoted from experimental.
• Time from report to tested rule: median time from a new threat report to a deployed, tested detection.
• Coverage claims verified: share of ATT&CK techniques marked covered that have at least one passing test.

Framework mapping

Common failure patterns

• Shipping generated rules straight to production because they "ran without errors", when they matched nothing.
• Marking ATT&CK techniques as covered from model-suggested tags without any test.
• Asking the model to translate every rule between SIEM dialects instead of using a deterministic converter, and losing case-insensitivity or null handling.
• Accepting model-proposed exclusions that remove the attack along with the noise.
• Pasting unredacted incident data, including live tokens and passwords, into an unapproved chatbot.
• Giving the assistant's service account write access to rules or playbooks.

Lessons from incidents

The EchoLeak vulnerability (CVE-2025-32711) in Microsoft 365 Copilot is not a SOC incident, but it shows the mechanism SOC teams must design against: a crafted email, retrieved later as ordinary context, carried instructions that made the assistant leak data it could access. An LLM that reads alert payloads, phishing emails or attacker command lines is exposed in the same way. Keep detection assistants read-only, scope their data access, and log what they retrieve and return.

Practitioner checklist

• Approve which models and assistants may see log data, and redact secrets before prompting.
• Provide schema, field lists and sample events in every rule-generation prompt.
• Enter all LLM-drafted rules as experimental with full Sigma metadata.
• Automate syntax, field and ATT&CK ID validation in CI and reject failures.
• Use deterministic converters for SIEM dialects.
• Require a passing true positive test and a benign baseline run for every new rule.
• Keep human review of detection logic mandatory and equal to human-written rules.
• Run the assistant under a read-only identity with no path to response actions.
• Track first-pass validation, review change rate and 30-day precision.

Standards and references

• SigmaHQ: Sigma Rules Specification
• SigmaHQ: pySigma
• NIST: AI 600-1 Generative Artificial Intelligence Profile
• OWASP: Top 10 for LLM Applications 2025
• MITRE ATT&CK
• Red Canary: Atomic Red Team
• Elastic: Detection Rules Repository
• Microsoft Learn: Security Copilot in Advanced Hunting

Related NHI Mgmt Group resources: Detection Engineering Guide · Sigma Rules Guide · Detection Engineering Maturity Model · Prompt Injection Guide