Detection Engineering Maturity Model

Most SOCs run on the detections their SIEM and EDR vendors shipped, and cannot say which attacks those rules would actually catch. Detection engineering turns detections into maintained software: written against a documented strategy, stored in version control, tested against real attack behaviour and retired when they stop earning their keep. This five-level model, developed by NHI Mgmt Group, describes how that practice matures from Ad hoc to Optimised across seven capability areas, including the identity and cloud detections that vendor content tends to miss. It is for detection engineers, SOC leads and security architects who need to assess where they stand and plan the next step. For the day-to-day practice itself, see the Detection Engineering Guide; for the wider SOC, see the SOC Maturity Model.

Key takeaways

The five levels are Ad hoc, Developing, Defined, Managed and Optimised, and most teams sit at Level 1 or 2: vendor rules, edited in the console, with no record of why they exist.
Detection as code is the hinge between Level 2 and Level 3. Rules in Git, written in a portable format such as Sigma, peer reviewed and deployed through a pipeline, are what make every later capability possible.
ATT&CK coverage percentage is a vanity metric. A green heatmap cell says a rule exists, not that it fires on the variants attackers use. Measure validated, robust coverage of the techniques in your threat model instead.
Untested detections are assumptions. Mature teams replay atomic tests or simulations against every production rule on a schedule and treat a silent rule as a defect.
Identity is the biggest blind spot. Service principals gaining new credentials, token replay and dormant service accounts waking up rarely appear in default rule packs, yet they are how cloud intrusions persist.

The five levels

Level	Name	What it looks like
1	Ad hoc	Vendor default rules only. Analysts tweak or disable noisy rules in the console with no record. Nobody owns detection content, and gaps are found during incidents.
2	Developing	A few custom rules exist, usually written after an incident or a threat report. Some are documented. ATT&CK tags appear, and leadership is shown a coverage heatmap. Testing is manual and occasional.
3	Defined	Detections live in version control in a standard format, each with a strategy document (for example the Palantir ADS template), an owner, a status and an ATT&CK mapping. Changes go through peer review and a deployment pipeline. A prioritised backlog is driven by a written threat model.
4	Managed	Every production rule has an automated test that runs in CI and on a schedule in production. Alert quality is measured per rule, and rules that fail thresholds are tuned or retired. Coverage is reported as validated technique coverage with robustness scores.
5	Optimised	Detection is a continuous loop fed by hunting, purple teaming, incidents and intelligence. Rules target behaviours core to a technique rather than attacker-controlled artefacts. Identity and cloud control planes are covered as thoroughly as endpoints, and response actions are wired to high-confidence rules.

Capability areas

Score each area separately. Progress is rarely even: a team may run excellent Sigma pipelines (Level 4 engineering) while still measuring success by heatmap colour (Level 2 measurement).

Capability area	Level 1 Ad hoc	Level 3 Defined	Level 5 Optimised
Detection as code	Rules edited in the SIEM console	Rules in Git, written in Sigma or the platform's query language with a schema, reviewed by pull request, deployed by pipeline with rollback	Content linted, converted (for example with pySigma or sigma-cli) and deployed to several backends from one source; changes traced to tickets and intelligence
Strategy and documentation	No record of intent	Each rule has goal, technical context, blind spots and assumptions, false positives, validation, priority and response notes	Documentation generated with the rule and reviewed at every change; blind spots feed the backlog
Threat-informed prioritisation	Whatever the vendor shipped	Backlog ranked from a threat model of relevant actors and techniques	Backlog re-ranked continuously from intelligence, hunting finds and incident lessons
Coverage measurement	Not measured	Techniques in scope listed; rules mapped with ATT&CK tags; telemetry gaps recorded	Validated coverage per technique and per behavioural variant, with robustness scored (for example using Summiting the Pyramid)
Testing and validation	None	Each rule tested by hand once before release with a documented true-positive sample	Automated tests per rule in CI and scheduled replay in production; failures raise tickets
Alert quality and lifecycle	Noisy rules disabled silently	Rule status (experimental, test, stable, deprecated) and severity used consistently; analyst feedback captured	Per-rule precision, volume and time to triage tracked; rules auto-flagged for tuning or retirement
Identity and cloud detections	Only IdP vendor alerts	Rules for cloud account misuse, new credentials on service principals and token anomalies	Behaviour baselines for each NHI and AI agent; detections cover control plane, SaaS and workload identity with automated containment

Measuring coverage without the vanity

A technique mapped to one rule turns green on a heatmap, but T1059.001 PowerShell alone has many execution paths, and a rule keyed on a filename or command-line string catches only one. MITRE's Center for Threat-Informed Defense addresses this directly: its Summiting the Pyramid project scores analytic robustness from Level 1 (Ephemeral) to Level 5 (Invariant Behaviors, Core to Technique), and its 2026 release adds implementation coverage (how many behaviourally distinct ways of performing a technique a rule observes) and detection quality. Report coverage only for techniques in your threat model, only where a test has confirmed the rule fires, and with the robustness level next to it.

Identity and NHI detections to add first

Behaviour	ATT&CK	Signal to build on
Valid cloud account used from a new location, ASN or client	T1078.004 Cloud Accounts	IdP sign-in logs; for NHIs, any interactive sign-in by a service account
New secret or certificate added to an app registration or service principal	T1098.001 Additional Cloud Credentials	Cloud directory audit logs, alerting on privileged applications first
OAuth token issued to an unexpected application or consent granted unusually	T1528 Steal Application Access Token	Consent and token issuance events
Access token replayed from infrastructure that did not request it	T1550.001 Application Access Token	Token identifiers or session IDs correlated with source IP and device

See the ITDR Guide and the Cloud Logging and Audit Trail Guide for the telemetry behind these rules, and the AI Agent Behaviour Baselining and Anomaly Detection Guide for agent identities.

Self-assessment questions

Can you list every production detection, its owner and the date it last fired on a confirmed true positive?
If a rule were deleted from the SIEM today, could you restore it, with its history, from version control in minutes?
Does each rule have a written strategy stating its blind spots and expected false positives?
Is your detection backlog ordered by a threat model, or by whoever asked last?
When you report ATT&CK coverage, does each covered technique have a passing test from the last 30 days?
How many of your rules match on attacker-controlled values (tool names, hashes, file paths) rather than behaviour?
Do you know the precision and weekly volume of your ten noisiest rules?
Would you detect a new client secret added to your most privileged service principal at 03:00 on a Sunday?
When telemetry from a log source stops, does a detection fire, or does coverage silently disappear?

Mostly "no" to questions 1 to 4 indicates Level 1 or 2. Confident "yes" through question 6 indicates Level 3. Evidence for all nine indicates Level 4 or above.

Moving up: roadmap

Level 1 to 2: inventory and own. Export every active rule, record owner and data source, and remove or document rules nobody can explain. Start capturing analyst feedback on false positives in the ticketing system.
Level 2 to 3: move to code. Pick one format (Sigma for portability, or your platform's language with a schema), put rules in Git, adopt a strategy template such as ADS, and require pull request review. Deploy through a pipeline that can roll back the previous version in one step. Write a threat model and re-rank the backlog from it.
Level 3 to 4: test and measure. Attach a test to each rule, for example an Atomic Red Team test mapped to the same ATT&CK technique or a recorded log sample, and run it in CI and on a schedule in a production-like environment. Track per-rule precision, volume and time to triage, and set retirement thresholds. Replace the heatmap with validated coverage. The BAS Guide covers tooling for scheduled replay.
Level 4 to 5: close the loop. Feed every hunt, incident and purple team exercise back into the backlog, raise robustness by rewriting rules against invariant behaviours, and wire high-confidence identity detections to automated containment such as session revocation. See the Purple Teaming Guide and the Threat Hunting Guide.

Metrics that show progress

Validated technique coverage: share of in-scope techniques with at least one rule that passed a test in the last 30 days.
Per-rule precision: true positives divided by total alerts over 90 days, with a retirement or tuning threshold.
Test pass rate: share of production rules whose scheduled test fired as expected.
Detection lead time: days from a new threat report or incident lesson to a deployed, tested rule.
Silent source alerts: number of log sources that stopped sending without a detection firing.

Framework mapping

Framework	Reference	Relevance
NIST CSF 2.0	DE.CM-01, DE.CM-03, DE.CM-09	Monitoring networks, personnel activity and computing environments for adverse events
NIST CSF 2.0	DE.AE-02, DE.AE-03, DE.AE-06	Analysis, correlation across sources and delivery of event information to staff and tools
NIST CSF 2.0	ID.IM-02	Improvements identified from security tests and exercises
NIST SP 800-53 Rev. 5	SI-4, AU-6, AU-12, CA-8	System monitoring, audit review and analysis, audit record generation, penetration testing

Practitioner checklist

Score all seven capability areas separately, using evidence from the repository and SIEM rather than policy documents.
Set Level 3 as the first target; it is the foundation every metric depends on.
Put every production rule in version control with an owner, status and strategy document.
Write a threat model and rank the detection backlog from it.
Stop reporting raw ATT&CK coverage percentage; report validated coverage with robustness.
Attach an automated test to each rule and run it on a schedule.
Retire rules that miss precision thresholds for two consecutive review periods.
Build the four identity detections above before adding more endpoint rules.

Standards and references

Related NHI Mgmt Group resources: Detection Engineering Guide · SOC Maturity Model · SIEM Guide · ITDR Guide

Detection Engineering Maturity Model: From Vendor Defaults to Detection as Code