Join our Newsletter — 33% off our NHI Course
Home› Guides› Detection Validation and Testing Guide: Prove Every Detection…
Guide SOC & Detection Engineering

Detection Validation and Testing Guide: Prove Every Detection Fires, and Keeps Firing

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 27 October 2026 16 min read
On this page

Most SOCs cannot say with evidence which of their detections would actually fire today, because rules are written once, deployed and never tested again. Log sources change format, agents stop reporting, parsers break after upgrades and a field rename silently turns a critical analytic into dead code. This guide covers how to prove that detections work: unit tests against recorded attack data, false positive tests against benign baselines, controlled emulation in production-like environments, and regression testing in a detection pipeline. It is written for detection engineers, SOC leads and purple team practitioners who already have a rule library and need a repeatable way to trust it.

Key takeaways

  • Test the whole chain, not just the rule logic. A detection only works if the telemetry is generated, collected, parsed, stored, matched and routed to a person. Each stage can fail on its own, so each needs its own test.
  • Every rule needs a true positive and a false positive test. Store a known-bad sample that must match and a benign baseline that must not, and keep both next to the rule in version control.
  • Use three layers of evidence. Replay recorded logs in CI for every change, run safe emulation (for example Atomic Red Team) on representative hosts on a schedule, and run full adversary emulation plans a few times a year.
  • Measure what is proven, not what is deployed. Report validated coverage per ATT&CK technique, time since last successful test and the share of rules with tests, rather than the number of rules switched on.
  • Treat a silent rule as an incident in the pipeline. Heartbeat and canary checks catch log sources and rules that stop producing data long before an attacker does.

What detection validation means

Detection validation is the practice of producing evidence that a specific analytic detects the behaviour it claims to detect, in your environment, with your telemetry, and that the resulting alert reaches the right queue. It is narrower than security control validation or breach and attack simulation, which test prevention and response controls broadly. See the Breach and Attack Simulation (BAS) Guide for the wider programme; this guide stays on the detection side.

A useful way to think about it is the detection chain. Each link is a separate failure point:

StageWhat can breakHow to test it
Telemetry generationAudit policy not set (for example process creation auditing off, Sysmon config missing a rule), agent uninstalled or in reduced modeEmulation on a representative host, then confirm the raw event exists on the endpoint
Collection and transportForwarder stopped, queue full, network path blocked, cloud diagnostic setting removedHeartbeat queries per source and per host; volume baselines
Parsing and normalisationVendor changes a field name or format, parser not updated, timestamps wrongParser unit tests against stored raw samples for each source version
Rule logicWrong field, case sensitivity, too narrow a match, exclusions that swallow real attacksReplay of recorded true positive data in CI; negative tests against benign baselines
Scheduling and scaleSearch window shorter than ingestion delay, query times out, rule disabled after a performance problemCompare expected run schedule with execution logs; alert on skipped runs
Alert routingAlert created but not sent to the case system, wrong severity, suppression too broadEnd-to-end test that checks the ticket or case exists with the right fields
Analyst responseRunbook missing, analyst closes as benignPurple team exercises with observed triage; review of closed tickets

Most mature teams find that more failures sit in collection, parsing and routing than in the rule logic itself. A test programme that only exercises rule logic will report false confidence.

Types of detection tests

No single technique proves everything. Combine cheap, frequent tests with expensive, realistic ones.

Test typeWhat it provesTypical frequencyCost and risk
Syntax and schema validationThe rule parses, required metadata (ATT&CK mapping, owner, severity) is present, IDs are uniqueEvery commitVery low
Unit test with recorded true positive dataRule logic matches a stored attack sampleEvery commit that touches the rule or its parserLow, needs a test SIEM or local engine
Negative (false positive) testRule does not match a benign baseline of normal activityEvery commitLow
Atomic emulation on a test hostTelemetry is generated and collected, and the deployed rule fires end to endWeekly to monthly per critical techniqueMedium, must be authorised and cleaned up
Adversary emulation planChains of behaviour across tactics are detected and correlated; analysts respondQuarterly to twice a yearHigh, needs planning and coordination
Health and canary checksSources and rules are still alive between testsContinuousVery low

Unit tests against recorded attack data

The fastest feedback loop is to replay stored logs of a known attack into a test instance and check that the rule matches. Public datasets make this possible without building a lab for every technique:

  • Splunk attack_data: curated attack log datasets organised by ATT&CK technique ID and then by the tool that produced them, such as atomic_red_team. Each dataset has a YAML file with metadata including mitre_technique, source and sourcetype, and the project notes that datasets can be replayed into streaming pipelines to validate detections in a production SIEM.
  • Splunk security_content and contentctl: detections carry a tests block that names a test, points attack_data at a dataset with its source and sourcetype, and sets test_type: unit. The contentctl test command supports modes all, selected and changes, the last of which tests only content changed against a target branch, which keeps CI fast.
  • SigmaHQ regression tests: Sigma rules can reference sample event logs through a regression_tests_path attribute; the project's test runner reads an info.yml with the expected match count and checks it with an EVTX checker. See the Sigma Rules Guide.
  • Elastic detection-rules: rules are stored as TOML, with validate-rule and validate-all commands for schema checks and a test command that runs unit tests over all rules in CI.

Whatever platform you use, the pattern is the same: one or more positive samples per rule, stored with the rule, replayed on every change. Record your own samples when you run emulation, because your log format, field mappings and enrichment differ from public datasets.

False positive testing against benign baselines

A rule that fires on everything is as useless as one that never fires. SigmaHQ runs "goodlog" tests in CI that execute its Windows rules against benign event log baselines (published as the evtx-baseline project, with Windows 7, Windows 10 and Windows 11 sets) and reports any match that is not a known false positive. Build the same thing internally: capture a week of normal logs from a gold build, admin workstations, servers and CI runners, scrub sensitive values, and make "zero new matches against baseline" a merge condition. Add samples of known noisy but legitimate behaviour, such as software deployment tools, backup agents and vulnerability scanners, because those are where rules typically fail in production.

Emulation in a representative environment

Replay tests prove logic but not telemetry. To prove that a host actually produces the event, run a controlled test of the technique on a machine built like production:

  • Atomic Red Team: a library of small tests mapped to ATT&CK, MIT licensed. Each technique folder holds a YAML definition with attack_technique, atomic_tests and, per test, supported_platforms, input_arguments, dependencies and an executor with command, cleanup_command and elevation_required. Every test has an auto_generated_guid, which is the identifier to record in your test results. The Invoke-AtomicRedTeam PowerShell module runs tests on Windows, macOS and Linux.
  • MITRE CTID Adversary Emulation Library: 11 full emulation plans for named groups (for example APT29, FIN7, Sandworm, Turla and Wizard Spider) and 12 micro emulation plans for compound behaviours such as log clearing, process injection, web shells and Active Directory enumeration. Plans have machine-readable steps that automation such as CALDERA can run. See the MITRE CALDERA Adversary Emulation Guide.
  • Safety rules: the Invoke-AtomicRedTeam documentation warns that tests can leave a system in an undesirable state, that you must have permission before testing, and that the test machine should resemble your build with EDR installed and checking in. Treat these as mandatory controls, not advice.

Designing tests that resist evasion

A rule that matches one tool's command line passes its unit test and still misses the next variant. Test design should push rules towards behaviour that attackers cannot easily change.

  • Test variants, not one sample: for each technique run several procedures. Atomic Red Team usually offers more than one test per technique; for LSASS memory access (T1003.001) the tests use different tools and DLL-based methods. A rule that catches only one is a signature, not a behavioural detection.
  • Score robustness: MITRE CTID's Summiting the Pyramid project ranks analytic robustness from level 1 (ephemeral values) to level 5 (invariant behaviours core to the technique), with visibility columns from application to kernel mode. Use the levels to decide which rules need more variant tests.
  • Change the inputs: rename binaries, change paths and parent processes, use encoded arguments for PowerShell (T1059.001), and run tests as different user types. If the rule breaks on a rename, it is matching an attacker-controlled value.
  • Test persistence techniques with cleanup: scheduled tasks (T1053.005) and Run keys (T1547.001) are common, cheap tests, but they change host state. Always run the cleanup command and verify it worked.
  • Test that defence impairment is seen: attackers who disable or modify tools (T1685) or the Windows event log (T1685.001) blind every other rule. These detections deserve the most frequent tests, ideally from a source the endpoint cannot switch off, such as the SIEM noticing the host went quiet.

Building a detection test pipeline

Validation scales only when it is part of how rules are shipped. See the Detection as Code Guide for repository structure; the steps below focus on the test stages.

  1. Inventory and prioritise. List rules with their ATT&CK technique, data source and owner. Pick the 20 to 50 techniques that matter most for your threat model (credential access, defence impairment, lateral movement, exfiltration from your crown-jewel systems) and start there.
  2. Add metadata and static checks. Require ATT&CK ID, data source, severity, owner, runbook link and test references in every rule file. Fail the build on missing fields or duplicate IDs, as SigmaHQ does in its own CI.
  3. Collect positive samples. Run the matching atomic test once on a lab host built like production, export the raw events, and store them with the rule. Use public datasets as a starting point where your format matches.
  4. Collect negative baselines. Export and scrub normal activity from representative systems. Refresh them after major OS, agent or application upgrades.
  5. Run replay tests in CI. On every pull request, test changed rules (and rules sharing a changed parser) against positive and negative samples in an ephemeral or dedicated test instance. Block merge on failure.
  6. Deploy in stages. Ship new or changed rules first as "monitor" or low-severity, compare alert volume with the test prediction for one to two weeks, then promote. Keep the previous version tagged so rollback is a single revert and redeploy.
  7. Schedule live emulation. Run automated atomic tests against a small, dedicated ring of production-like hosts on a schedule, and check that each expected alert exists in the case system within the expected time. Record test GUID, host, time, expected rule and result.
  8. Run periodic emulation plans with the SOC. Two to four times a year run a full or micro emulation plan as a purple team exercise. See the Purple Teaming Guide.
  9. Feed results back. Every failed test becomes a ticket with an owner: fix telemetry, parser, rule or routing, then re-test. Track this as engineering debt.

Rollback and safety: keep emulation hosts in their own group with tagged assets so alerts can be correlated to tests and are not escalated as real incidents, but never suppress them so broadly that a real attacker on the same host would be hidden. Announce test windows to the SOC lead only, not the whole team, if you also want to measure analyst response.

Decision guidance

SituationPreferWhy
Small team, one SIEM, few engineersReplay tests for the top 20 techniques plus monthly manual atomic runsHighest coverage gain per hour; no extra platform needed
Large SOC with detection as codeFull CI with positive and negative samples, scheduled automated emulation, health monitoring per sourceRule volume is too high for manual checks; changes are frequent
Heavily regulated or high-threat sectorAdd threat-led emulation plans and external red team exercisesSupervisors and boards ask for evidence against realistic actors
Commercial BAS platform already licensedUse it for scheduled live tests, keep replay tests in your own pipelineBAS proves end-to-end outcomes; replay tests catch logic regressions before deployment
Cloud and SaaS detectionsReplay of stored audit log samples, plus safe actions by a dedicated test identityMany cloud techniques cannot be run safely on shared tenants; test identities must be tightly scoped

Where identity fits

Many of the highest-value detections are about identities: service accounts logging on interactively, API keys used from new locations, OAuth grants added to sensitive applications. Testing them needs a dedicated test identity, which is itself a non-human identity with real permissions. Give it the minimum rights needed, store its credential in a vault, rotate it, tag it clearly in the identity provider so its activity can be correlated with tests, and alert if it is ever used outside the test window. An unmanaged "purple team" account with broad rights is a common audit finding. See the Detection Engineering for Non-Human Identities Guide.

Detection and measurement

Validation itself needs monitoring. Log every test run with test ID, technique, target, expected detection, observed alert and time to alert, so results can be reported and trended.

  • Validated coverage: share of priority ATT&CK techniques with at least one detection that passed a live test in the last 90 days. Report it separately from "rules deployed".
  • Test coverage of the rule library: share of production rules with both a positive and a negative test in version control. Aim for all critical and high severity rules first.
  • Test pass rate and failure causes: percentage of live tests that produced the expected alert, broken down by cause (telemetry, parser, logic, routing). The breakdown tells you where to invest.
  • Time to alert: median time from test execution to alert in the case system, compared with your detection objectives.
  • Silent source and silent rule count: number of log sources or rules with zero events or zero runs over their expected window, from heartbeat checks.
  • Mean age of last successful test: days since each critical rule last passed. Rising age means validation is not keeping up with change.

Common failure patterns

  • Testing only the rule, never the telemetry. Replay tests pass while the production agent on half the estate has been in reduced mode for months. Live emulation and heartbeat checks are what catch this.
  • One sample per rule. The rule matches the exact command line of the test tool and nothing else. Add variant tests and use robustness scoring.
  • Exclusions that swallow attacks. A broad exclusion added to stop noise (for example everything under an admin tools path) also excludes the attacker who copies a tool there. Negative tests should be specific, and exclusions reviewed with a positive test that runs from the excluded path.
  • Emulation without cleanup or authorisation. Tests leave scheduled tasks, dumps or registry keys behind, or run on systems the owner did not approve. Use written authorisation, a defined host ring and verified cleanup.
  • Suppressing test alerts too broadly. Teams filter all alerts from test hosts, which hides real activity on those hosts and makes the test meaningless for routing. Correlate on test ID, not on host alone.
  • No owner for failed tests. Results land in a report nobody acts on. Every failure needs a ticket, an owner and a re-test date.

Framework mapping

FrameworkReferenceRelevance to detection validation
NIST SP 800-53 Rev. 5SI-4 System Monitoring, SI-4(2) Automated Tools and Mechanisms for Real-time AnalysisThe monitoring capability being validated
NIST SP 800-53 Rev. 5CA-2 Control Assessments, CA-7 Continuous MonitoringAssessing that monitoring controls operate as intended, on an ongoing basis
NIST SP 800-53 Rev. 5CA-8 Penetration Testing, CA-8(2) Red Team ExercisesEmulation and red team exercises that exercise detections
NIST SP 800-53 Rev. 5AU-6 Audit Record Review, Analysis, and ReportingReview and analysis of the records that detections depend on
NIST CSF 2.0DE.CM-01, DE.CM-09, DE.AE-02, PR.PS-04Monitoring of networks and computing, analysis of adverse events, log generation
NIST CSF 2.0ID.IM-02Improvements identified from security tests and exercises
CIS Controls v8.1Control 8 Audit Log Management; Control 18 Penetration TestingLogging that detections rely on; testing that validates them
ISO/IEC 27001:2022 Annex AA.8.15 Logging; A.8.16 Monitoring activitiesLogging and monitoring controls whose effectiveness is evidenced by tests
MITRE ATT&CK v20Technique IDs used as the test catalogueCommon language for coverage and test results

Practitioner checklist

  • Map every production rule to an ATT&CK technique, a data source, an owner and a runbook.
  • Store at least one positive sample and one benign baseline per critical rule, next to the rule in version control.
  • Run syntax, schema and replay tests in CI on every rule or parser change, and block merges on failure.
  • Maintain a benign baseline from representative systems and refresh it after major upgrades.
  • Run authorised atomic tests on a dedicated ring of production-like hosts on a schedule, with verified cleanup.
  • Check end to end: the alert must reach the case system with correct severity and fields.
  • Run heartbeat checks for every log source and alert on rules that skip runs or return nothing for too long.
  • Test each technique with several procedure variants and score robustness for critical rules.
  • Run at least two purple team emulation exercises a year and turn every failure into an owned ticket.
  • Report validated coverage and test age, not rule count.

Standards and references

Related NHI Mgmt Group resources: Detection Engineering Guide · Breach and Attack Simulation (BAS) Guide · Detection Rule Documentation Template · Detection Engineering Maturity Model

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 27 October 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.