Most SOCs cannot say with evidence which of their detections would actually fire today, because rules are written once, deployed and never tested again. Log sources change format, agents stop reporting, parsers break after upgrades and a field rename silently turns a critical analytic into dead code. This guide covers how to prove that detections work: unit tests against recorded attack data, false positive tests against benign baselines, controlled emulation in production-like environments, and regression testing in a detection pipeline. It is written for detection engineers, SOC leads and purple team practitioners who already have a rule library and need a repeatable way to trust it.
Key takeaways
- Test the whole chain, not just the rule logic. A detection only works if the telemetry is generated, collected, parsed, stored, matched and routed to a person. Each stage can fail on its own, so each needs its own test.
- Every rule needs a true positive and a false positive test. Store a known-bad sample that must match and a benign baseline that must not, and keep both next to the rule in version control.
- Use three layers of evidence. Replay recorded logs in CI for every change, run safe emulation (for example Atomic Red Team) on representative hosts on a schedule, and run full adversary emulation plans a few times a year.
- Measure what is proven, not what is deployed. Report validated coverage per ATT&CK technique, time since last successful test and the share of rules with tests, rather than the number of rules switched on.
- Treat a silent rule as an incident in the pipeline. Heartbeat and canary checks catch log sources and rules that stop producing data long before an attacker does.
What detection validation means
Detection validation is the practice of producing evidence that a specific analytic detects the behaviour it claims to detect, in your environment, with your telemetry, and that the resulting alert reaches the right queue. It is narrower than security control validation or breach and attack simulation, which test prevention and response controls broadly. See the Breach and Attack Simulation (BAS) Guide for the wider programme; this guide stays on the detection side.
A useful way to think about it is the detection chain. Each link is a separate failure point:
| Stage | What can break | How to test it |
|---|---|---|
| Telemetry generation | Audit policy not set (for example process creation auditing off, Sysmon config missing a rule), agent uninstalled or in reduced mode | Emulation on a representative host, then confirm the raw event exists on the endpoint |
| Collection and transport | Forwarder stopped, queue full, network path blocked, cloud diagnostic setting removed | Heartbeat queries per source and per host; volume baselines |
| Parsing and normalisation | Vendor changes a field name or format, parser not updated, timestamps wrong | Parser unit tests against stored raw samples for each source version |
| Rule logic | Wrong field, case sensitivity, too narrow a match, exclusions that swallow real attacks | Replay of recorded true positive data in CI; negative tests against benign baselines |
| Scheduling and scale | Search window shorter than ingestion delay, query times out, rule disabled after a performance problem | Compare expected run schedule with execution logs; alert on skipped runs |
| Alert routing | Alert created but not sent to the case system, wrong severity, suppression too broad | End-to-end test that checks the ticket or case exists with the right fields |
| Analyst response | Runbook missing, analyst closes as benign | Purple team exercises with observed triage; review of closed tickets |
Most mature teams find that more failures sit in collection, parsing and routing than in the rule logic itself. A test programme that only exercises rule logic will report false confidence.
Types of detection tests
No single technique proves everything. Combine cheap, frequent tests with expensive, realistic ones.
| Test type | What it proves | Typical frequency | Cost and risk |
|---|---|---|---|
| Syntax and schema validation | The rule parses, required metadata (ATT&CK mapping, owner, severity) is present, IDs are unique | Every commit | Very low |
| Unit test with recorded true positive data | Rule logic matches a stored attack sample | Every commit that touches the rule or its parser | Low, needs a test SIEM or local engine |
| Negative (false positive) test | Rule does not match a benign baseline of normal activity | Every commit | Low |
| Atomic emulation on a test host | Telemetry is generated and collected, and the deployed rule fires end to end | Weekly to monthly per critical technique | Medium, must be authorised and cleaned up |
| Adversary emulation plan | Chains of behaviour across tactics are detected and correlated; analysts respond | Quarterly to twice a year | High, needs planning and coordination |
| Health and canary checks | Sources and rules are still alive between tests | Continuous | Very low |
Unit tests against recorded attack data
The fastest feedback loop is to replay stored logs of a known attack into a test instance and check that the rule matches. Public datasets make this possible without building a lab for every technique:
- Splunk attack_data: curated attack log datasets organised by ATT&CK technique ID and then by the tool that produced them, such as
atomic_red_team. Each dataset has a YAML file with metadata includingmitre_technique,sourceandsourcetype, and the project notes that datasets can be replayed into streaming pipelines to validate detections in a production SIEM. - Splunk security_content and contentctl: detections carry a
testsblock that names a test, pointsattack_dataat a dataset with itssourceandsourcetype, and setstest_type: unit. Thecontentctl testcommand supports modesall,selectedandchanges, the last of which tests only content changed against a target branch, which keeps CI fast. - SigmaHQ regression tests: Sigma rules can reference sample event logs through a
regression_tests_pathattribute; the project's test runner reads aninfo.ymlwith the expected match count and checks it with an EVTX checker. See the Sigma Rules Guide. - Elastic detection-rules: rules are stored as TOML, with
validate-ruleandvalidate-allcommands for schema checks and atestcommand that runs unit tests over all rules in CI.
Whatever platform you use, the pattern is the same: one or more positive samples per rule, stored with the rule, replayed on every change. Record your own samples when you run emulation, because your log format, field mappings and enrichment differ from public datasets.
False positive testing against benign baselines
A rule that fires on everything is as useless as one that never fires. SigmaHQ runs "goodlog" tests in CI that execute its Windows rules against benign event log baselines (published as the evtx-baseline project, with Windows 7, Windows 10 and Windows 11 sets) and reports any match that is not a known false positive. Build the same thing internally: capture a week of normal logs from a gold build, admin workstations, servers and CI runners, scrub sensitive values, and make "zero new matches against baseline" a merge condition. Add samples of known noisy but legitimate behaviour, such as software deployment tools, backup agents and vulnerability scanners, because those are where rules typically fail in production.
Emulation in a representative environment
Replay tests prove logic but not telemetry. To prove that a host actually produces the event, run a controlled test of the technique on a machine built like production:
- Atomic Red Team: a library of small tests mapped to ATT&CK, MIT licensed. Each technique folder holds a YAML definition with
attack_technique,atomic_testsand, per test,supported_platforms,input_arguments,dependenciesand anexecutorwithcommand,cleanup_commandandelevation_required. Every test has anauto_generated_guid, which is the identifier to record in your test results. The Invoke-AtomicRedTeam PowerShell module runs tests on Windows, macOS and Linux. - MITRE CTID Adversary Emulation Library: 11 full emulation plans for named groups (for example APT29, FIN7, Sandworm, Turla and Wizard Spider) and 12 micro emulation plans for compound behaviours such as log clearing, process injection, web shells and Active Directory enumeration. Plans have machine-readable steps that automation such as CALDERA can run. See the MITRE CALDERA Adversary Emulation Guide.
- Safety rules: the Invoke-AtomicRedTeam documentation warns that tests can leave a system in an undesirable state, that you must have permission before testing, and that the test machine should resemble your build with EDR installed and checking in. Treat these as mandatory controls, not advice.
Designing tests that resist evasion
A rule that matches one tool's command line passes its unit test and still misses the next variant. Test design should push rules towards behaviour that attackers cannot easily change.
- Test variants, not one sample: for each technique run several procedures. Atomic Red Team usually offers more than one test per technique; for LSASS memory access (T1003.001) the tests use different tools and DLL-based methods. A rule that catches only one is a signature, not a behavioural detection.
- Score robustness: MITRE CTID's Summiting the Pyramid project ranks analytic robustness from level 1 (ephemeral values) to level 5 (invariant behaviours core to the technique), with visibility columns from application to kernel mode. Use the levels to decide which rules need more variant tests.
- Change the inputs: rename binaries, change paths and parent processes, use encoded arguments for PowerShell (T1059.001), and run tests as different user types. If the rule breaks on a rename, it is matching an attacker-controlled value.
- Test persistence techniques with cleanup: scheduled tasks (T1053.005) and Run keys (T1547.001) are common, cheap tests, but they change host state. Always run the cleanup command and verify it worked.
- Test that defence impairment is seen: attackers who disable or modify tools (T1685) or the Windows event log (T1685.001) blind every other rule. These detections deserve the most frequent tests, ideally from a source the endpoint cannot switch off, such as the SIEM noticing the host went quiet.
Building a detection test pipeline
Validation scales only when it is part of how rules are shipped. See the Detection as Code Guide for repository structure; the steps below focus on the test stages.
- Inventory and prioritise. List rules with their ATT&CK technique, data source and owner. Pick the 20 to 50 techniques that matter most for your threat model (credential access, defence impairment, lateral movement, exfiltration from your crown-jewel systems) and start there.
- Add metadata and static checks. Require ATT&CK ID, data source, severity, owner, runbook link and test references in every rule file. Fail the build on missing fields or duplicate IDs, as SigmaHQ does in its own CI.
- Collect positive samples. Run the matching atomic test once on a lab host built like production, export the raw events, and store them with the rule. Use public datasets as a starting point where your format matches.
- Collect negative baselines. Export and scrub normal activity from representative systems. Refresh them after major OS, agent or application upgrades.
- Run replay tests in CI. On every pull request, test changed rules (and rules sharing a changed parser) against positive and negative samples in an ephemeral or dedicated test instance. Block merge on failure.
- Deploy in stages. Ship new or changed rules first as "monitor" or low-severity, compare alert volume with the test prediction for one to two weeks, then promote. Keep the previous version tagged so rollback is a single revert and redeploy.
- Schedule live emulation. Run automated atomic tests against a small, dedicated ring of production-like hosts on a schedule, and check that each expected alert exists in the case system within the expected time. Record test GUID, host, time, expected rule and result.
- Run periodic emulation plans with the SOC. Two to four times a year run a full or micro emulation plan as a purple team exercise. See the Purple Teaming Guide.
- Feed results back. Every failed test becomes a ticket with an owner: fix telemetry, parser, rule or routing, then re-test. Track this as engineering debt.
Rollback and safety: keep emulation hosts in their own group with tagged assets so alerts can be correlated to tests and are not escalated as real incidents, but never suppress them so broadly that a real attacker on the same host would be hidden. Announce test windows to the SOC lead only, not the whole team, if you also want to measure analyst response.
Decision guidance
| Situation | Prefer | Why |
|---|---|---|
| Small team, one SIEM, few engineers | Replay tests for the top 20 techniques plus monthly manual atomic runs | Highest coverage gain per hour; no extra platform needed |
| Large SOC with detection as code | Full CI with positive and negative samples, scheduled automated emulation, health monitoring per source | Rule volume is too high for manual checks; changes are frequent |
| Heavily regulated or high-threat sector | Add threat-led emulation plans and external red team exercises | Supervisors and boards ask for evidence against realistic actors |
| Commercial BAS platform already licensed | Use it for scheduled live tests, keep replay tests in your own pipeline | BAS proves end-to-end outcomes; replay tests catch logic regressions before deployment |
| Cloud and SaaS detections | Replay of stored audit log samples, plus safe actions by a dedicated test identity | Many cloud techniques cannot be run safely on shared tenants; test identities must be tightly scoped |
Where identity fits
Many of the highest-value detections are about identities: service accounts logging on interactively, API keys used from new locations, OAuth grants added to sensitive applications. Testing them needs a dedicated test identity, which is itself a non-human identity with real permissions. Give it the minimum rights needed, store its credential in a vault, rotate it, tag it clearly in the identity provider so its activity can be correlated with tests, and alert if it is ever used outside the test window. An unmanaged "purple team" account with broad rights is a common audit finding. See the Detection Engineering for Non-Human Identities Guide.
Detection and measurement
Validation itself needs monitoring. Log every test run with test ID, technique, target, expected detection, observed alert and time to alert, so results can be reported and trended.
- Validated coverage: share of priority ATT&CK techniques with at least one detection that passed a live test in the last 90 days. Report it separately from "rules deployed".
- Test coverage of the rule library: share of production rules with both a positive and a negative test in version control. Aim for all critical and high severity rules first.
- Test pass rate and failure causes: percentage of live tests that produced the expected alert, broken down by cause (telemetry, parser, logic, routing). The breakdown tells you where to invest.
- Time to alert: median time from test execution to alert in the case system, compared with your detection objectives.
- Silent source and silent rule count: number of log sources or rules with zero events or zero runs over their expected window, from heartbeat checks.
- Mean age of last successful test: days since each critical rule last passed. Rising age means validation is not keeping up with change.
Common failure patterns
- Testing only the rule, never the telemetry. Replay tests pass while the production agent on half the estate has been in reduced mode for months. Live emulation and heartbeat checks are what catch this.
- One sample per rule. The rule matches the exact command line of the test tool and nothing else. Add variant tests and use robustness scoring.
- Exclusions that swallow attacks. A broad exclusion added to stop noise (for example everything under an admin tools path) also excludes the attacker who copies a tool there. Negative tests should be specific, and exclusions reviewed with a positive test that runs from the excluded path.
- Emulation without cleanup or authorisation. Tests leave scheduled tasks, dumps or registry keys behind, or run on systems the owner did not approve. Use written authorisation, a defined host ring and verified cleanup.
- Suppressing test alerts too broadly. Teams filter all alerts from test hosts, which hides real activity on those hosts and makes the test meaningless for routing. Correlate on test ID, not on host alone.
- No owner for failed tests. Results land in a report nobody acts on. Every failure needs a ticket, an owner and a re-test date.
Framework mapping
| Framework | Reference | Relevance to detection validation |
|---|---|---|
| NIST SP 800-53 Rev. 5 | SI-4 System Monitoring, SI-4(2) Automated Tools and Mechanisms for Real-time Analysis | The monitoring capability being validated |
| NIST SP 800-53 Rev. 5 | CA-2 Control Assessments, CA-7 Continuous Monitoring | Assessing that monitoring controls operate as intended, on an ongoing basis |
| NIST SP 800-53 Rev. 5 | CA-8 Penetration Testing, CA-8(2) Red Team Exercises | Emulation and red team exercises that exercise detections |
| NIST SP 800-53 Rev. 5 | AU-6 Audit Record Review, Analysis, and Reporting | Review and analysis of the records that detections depend on |
| NIST CSF 2.0 | DE.CM-01, DE.CM-09, DE.AE-02, PR.PS-04 | Monitoring of networks and computing, analysis of adverse events, log generation |
| NIST CSF 2.0 | ID.IM-02 | Improvements identified from security tests and exercises |
| CIS Controls v8.1 | Control 8 Audit Log Management; Control 18 Penetration Testing | Logging that detections rely on; testing that validates them |
| ISO/IEC 27001:2022 Annex A | A.8.15 Logging; A.8.16 Monitoring activities | Logging and monitoring controls whose effectiveness is evidenced by tests |
| MITRE ATT&CK v20 | Technique IDs used as the test catalogue | Common language for coverage and test results |
Practitioner checklist
- Map every production rule to an ATT&CK technique, a data source, an owner and a runbook.
- Store at least one positive sample and one benign baseline per critical rule, next to the rule in version control.
- Run syntax, schema and replay tests in CI on every rule or parser change, and block merges on failure.
- Maintain a benign baseline from representative systems and refresh it after major upgrades.
- Run authorised atomic tests on a dedicated ring of production-like hosts on a schedule, with verified cleanup.
- Check end to end: the alert must reach the case system with correct severity and fields.
- Run heartbeat checks for every log source and alert on rules that skip runs or return nothing for too long.
- Test each technique with several procedure variants and score robustness for critical rules.
- Run at least two purple team emulation exercises a year and turn every failure into an owned ticket.
- Report validated coverage and test age, not rule count.
Standards and references
- NIST: SP 800-53 Rev. 5 Security and Privacy Controls
- NIST: Cybersecurity Framework 2.0
- MITRE ATT&CK: Enterprise Matrix
- MITRE Center for Threat-Informed Defense: Adversary Emulation Library
- MITRE Center for Threat-Informed Defense: Summiting the Pyramid
- Red Canary: Atomic Red Team
- Splunk: attack_data repository
- SigmaHQ: Sigma rule repository
Related NHI Mgmt Group resources: Detection Engineering Guide · Breach and Attack Simulation (BAS) Guide · Detection Rule Documentation Template · Detection Engineering Maturity Model