ITDR validation: Proving Identity Detections Work, Safely

Most organisations run identity threat detection without evidence that the detections fire, reach an analyst and lead to action when an identity is actually misused. Rules drift as directories, identity providers and log pipelines change, and a silent detection looks the same as a quiet week. This guide sets out a defensive validation programme for identity threat detection and response (ITDR): a coverage map against MITRE ATT&CK identity techniques, controlled tests described in terms of what to test and what evidence to collect, measurement of alert fidelity and response time, and reporting. It is for detection engineers, identity security teams and SOC leads. It assumes the detections described in the ITDR Guide and does not repeat them. It contains no attack commands or tooling steps by design.

Key takeaways

• Start with a coverage map, not a test. List the identity techniques that matter to you, the telemetry each needs and the detection that should fire, then test against that map.
• Test the whole chain. A pass means the event was logged, the rule fired, the alert reached the right queue with enough context, and an analyst could act on it.
• Control every test. Use dedicated test identities, written approval, a change window, a stop condition and a rollback plan, and tell the people who need to know.
• Measure fidelity and speed. Track detection rate per technique, false positive rate, time to alert and time to triage, and trend them over quarters.
• Report gaps as work items. Each failed test becomes a ticket with an owner: missing telemetry, broken rule, noisy rule or slow response.

Why ITDR detections need validation

Identity detections fail quietly for ordinary reasons: a domain controller stops forwarding its Security log, an identity provider changes an event schema, a rule is tuned so tightly it never fires, or alerts land in a queue nobody watches. NIST SP 800-53 expects organisations to test incident response capability (IR-3), assess controls (CA-2) and monitor continuously (CA-7). NIST CSF 2.0 asks that improvements be identified from security tests and exercises (ID.IM-02). Validation turns those expectations into evidence for identity detection specifically.

Validation sits alongside, not instead of, related practices. Threat hunting looks for activity your rules miss (see the Identity Threat Hunting Guide); purple teaming tests detections collaboratively with an offensive team (see the Purple Teaming Guide); breach and attack simulation automates repeatable checks (see the Breach and Attack Simulation (BAS) Guide). An ITDR validation programme decides what must be proven, how often, and how results are measured.

Building the detection coverage map

Build the map in a spreadsheet or your detection repository. One row per technique, with the telemetry it depends on and the detection you expect to fire. The MITRE ATT&CK Coverage Mapping Guide covers the general method; this table shows identity-focused rows (technique IDs checked against ATT&CK v20).

For each row, record four states: telemetry present (yes, partial, no), rule exists, last validated date, and result. Rows with no telemetry are not test candidates yet: they are logging work. The Active Directory Security Monitoring and Event IDs Guide lists the directory audit settings most rows depend on. Include non-human identities: service accounts, workload identities and OAuth apps belong on the map as much as people (see the Detection Engineering for Non-Human Identities Guide).

Controlled test design

Each test answers one question: when this behaviour happens to this identity, do we see it and act? Write tests in terms of the behaviour and evidence, not the tooling. Every test plan should contain:

• Scope: the technique row, the systems in scope and those explicitly out of scope (for example production domain controllers outside the window).
• Test identities: dedicated, clearly named test accounts and service accounts with no real access to data, created through normal provisioning, recorded in the identity inventory and disabled between tests. Never test with real user or executive accounts.
• Approvals: written approval from the system owner and security leadership, and agreement from the SOC on whether the test is announced or unannounced to analysts.
• Change window: a scheduled window recorded in change management, with the identity, network and SOC on-call contacts named.
• Safety limits: rate limits that stay well under lockout thresholds, no changes to real privileged groups, and a stop condition (any production impact, any unexpected account lockout, any real alert from another source).
• Rollback: how each change is reversed, for example removing the test account from a group, revoking a test OAuth consent, resetting credentials and disabling the account afterwards.
• Evidence to collect: the raw event IDs and timestamps at source, the time the event arrived in the SIEM, the alert ID and time, the queue it reached, the analyst's triage notes and the time of the first response action.

Use honey identities to make some tests cleaner: a decoy service account that no legitimate process should touch gives a high-fidelity signal for tests and for real activity (see Decoy accounts in Active Directory). If an automated simulation platform runs the behaviour, apply the same approvals and limits, and keep its own credentials in the identity inventory. For tests that need offensive expertise, use an authorised purple team engagement rather than ad hoc testing.

Test types and when to use them

Small organisations can start with log presence checks and rule unit tests, then a few quarterly controlled tests for password spraying, MFA prompt floods and privileged group changes. Large organisations with several directories and identity providers should automate presence checks per source and run controlled tests per environment, because coverage often differs between forests and tenants.

Measuring alert fidelity and response time

These metrics are our recommendations. Keep them per technique row so trends are visible.

Fidelity needs both sides: a rule that always fires in tests but floods analysts with benign alerts is not working. Pair test results with 30 days of production alert outcomes before you call a row green.

Implementation roadmap

1. Weeks 1 to 2: agree scope, owners and approval process; build the coverage map for 10 to 15 priority rows.
2. Weeks 3 to 4: automate log presence checks for every identity source and fix missing telemetry.
3. Weeks 5 to 8: add recorded-event unit tests for each priority rule in your detection pipeline.
4. Quarter 2: run the first controlled behaviour tests with test identities in change windows; record evidence and metrics.
5. Quarter 3 onwards: add a purple team exercise, an unannounced response test, and re-test after every major identity platform change.

Roll back by disabling any test identity immediately after each window, revoking any test consents, and confirming in the identity inventory that nothing created for the test remains active.

Reporting results

• Coverage heat map: technique rows coloured by last result and freshness, for the SOC and identity teams.
• Gap register: each failure classified as missing telemetry, rule logic, enrichment, routing or response, with an owner and due date.
• Trend summary for leadership: detection rate, time to alert and validation freshness by quarter, plus the top three gaps and their status.
• Evidence pack: approvals, change records, test identity lifecycle records and raw evidence, retained for audit and kept with incident response test records.

Common failure patterns

• Testing the rule, not the chain: a rule passes in a lab while the production log source has been silent for weeks.
• Using real accounts: tests run with a real admin's account, causing lockouts and polluting the audit trail.
• Leftover test artefacts: test accounts, group memberships or consents left active after the window, creating the exposure the test was meant to catch.
• Cloud and NHI blind spots: the map covers on-premises directory techniques but not identity provider configuration changes, OAuth grants or service accounts.
• No re-test after change: a directory migration or IdP change silently breaks parsing, and nobody re-validates.
• Results without owners: gaps reported in slides but never ticketed.

Framework mapping

Practitioner checklist

• Build a coverage map of priority identity techniques with telemetry, rule and last validation date.
• Automate log presence checks for every directory, IdP and MFA source.
• Add recorded-event unit tests to every identity rule change.
• Create dedicated, inventoried test identities and disable them between windows.
• Require written approval, a change window, a stop condition and a rollback plan for every live test.
• Collect source event times, SIEM arrival, alert time, queue and first response for every test.
• Track detection rate, time to alert, false positive rate and validation freshness per row.
• Include service accounts, OAuth apps and IdP configuration changes on the map.
• Re-validate after every major identity platform or logging change.
• Ticket every gap with an owner and report trends to leadership quarterly.

Standards and references

• MITRE: ATT&CK Enterprise Matrix
• NIST: SP 800-53 Rev. 5 Security and Privacy Controls
• NIST: Cybersecurity Framework 2.0
• NIST: SP 800-84 Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities
• NIST: SP 800-115 Technical Guide to Information Security Testing and Assessment
• MITRE ATT&CK: T1078 Valid Accounts

Related NHI Mgmt Group resources: ITDR Guide · MITRE ATT&CK Coverage Mapping Guide · Purple Teaming Guide · Detection Engineering Guide