TL;DR: Continuous purple teaming only works when validation is automated, procedure-level tracking is precise, and detection engineering is treated like software, according to Sprocket Security’s interview with Northwestern Mutual’s Gary Lobermier. The operational lesson is that manual exercises and static coverage claims quickly decay as environments, tools, and attacker techniques change.
At a glance
What this is: This episode argues that purple teaming only scales when validation is automated, procedure-level tracking is precise, and detection engineering is treated like software.
Why it matters: For IAM, NHI, and broader security teams, the lesson is that control assurance decays unless testing keeps pace with changing environments, tools, and adversary behaviour.
Context
Purple teaming is the practice of continuously testing defensive controls against realistic attacker behaviours, then feeding the results back into detection engineering. In this article, the security gap is not lack of tools, but the mismatch between static validation and environments that change daily across endpoints, cloud instances, and operating systems.
The identity connection is indirect but real: when detection and response rely on access controls, EDR coverage, and environment-specific rules, missed validation can leave service accounts, admin paths, and cloud workloads less observable than teams assume. The article frames scaling as an engineering problem because manual tests and spreadsheet tracking cannot keep pace with operational drift.
Key questions
Q: What breaks when purple team validation is still manual?
A: Manual purple teaming breaks when the environment changes faster than the test cycle. A one-off exercise can show that a detection worked on a specific day, but it cannot prove that the same control still works after new assets, telemetry changes, or EDR updates. The result is coverage theatre, where confidence rises faster than assurance.
Q: Why does procedure-level tracking matter in detection engineering?
A: Procedure-level tracking matters because attackers do not use techniques in the abstract, they use specific implementation variants. If a team only records the technique name, it can overstate coverage and miss the exact behavior its detection logic was built to catch. Granularity is what turns a validation program into evidence, not just a report.
Q: What are the signs that purple team coverage is failing?
A: Common signs include repeated passes on high-level techniques, stale test results after infrastructure changes, and detection gaps that keep reappearing in the same control area. If the dashboard says coverage is complete but the environment keeps changing, the program is probably measuring activity more reliably than resilience.
A: Security teams should use continuous attack simulation to test controls end to end, not as isolated point checks. The goal is to see how email security, perimeter defenses, detection logic, and response workflows behave together under realistic attack paths. That approach exposes control gaps faster, shows where a single weakness cascades, and helps prioritise remediation based on real exposure across the kill chain.
Technical breakdown
Why continuous purple team validation needs automation
Manual adversary emulation works for isolated exercises, but it does not scale to daily assurance across heterogeneous environments. Continuous purple teaming turns validation into a repeatable pipeline: schedule tests, execute them across target systems, capture outcomes, and compare expected versus actual detections. That changes the problem from a point-in-time assessment into a control feedback loop. The key insight is that the unit of value is not a report, it is a current answer to whether a specific detection still works after environment changes, agent updates, or configuration drift.
Practical implication: Treat validation runs like production jobs with repeatable inputs, logged outputs, and change correlation.
What the MITRE ATT&CK procedure gap means for coverage
MITRE ATT&CK is strong on techniques and sub-techniques, but real intrusions are executed through procedures, the implementation-level variants that adversaries actually use. A team can claim it tested process injection and still miss most practical variants if it does not track which procedure was exercised. That creates a coverage illusion. Granular procedure mapping matters because detection logic is often tuned to a specific behaviour pattern, not the abstract technique label. Without that level of tracking, purple team metrics can look complete while leaving real bypass paths untested.
Practical implication: Track procedure variants separately from techniques so detection coverage reflects the behavior your controls actually need to catch.
Why EDR validation must be repeated, not assumed
EDR tools are not static guarantees. Detection logic, environment composition, and attacker tradecraft all change, so a rule that worked last quarter may fail today. Annual or ad hoc testing cannot keep pace with that drift, especially when teams deploy new agents, update configurations, or change telemetry sources. Continuous validation exposes whether the control still produces signal under current conditions, which is more useful than arguing about product capability in the abstract. The broader technical point is that detection quality is a property of the full stack, not the brand name of the endpoint tool.
Practical implication: Re-test EDR detections whenever the environment or telemetry path changes, not just on an audit calendar.
NHI Mgmt Group analysis
Continuous validation is becoming a control discipline, not a red-team exercise. The article shows that purple teaming only becomes operationally useful when it is treated like software testing with repeatable execution, versioned inputs, and measurable outcomes. That shifts the governance question from whether a test was run to whether the control still works after change. Practitioners should measure detection assurance as a living control state, not an annual event.
The procedure gap is the real source of false confidence. Technique-level reporting can overstate coverage when teams do not track the specific procedure variants that adversaries actually use. That creates a governance blind spot because management dashboards can look complete while practical detection remains partial. Practitioners should insist on procedure granularity wherever attack simulation feeds detection engineering.
Coverage theater: the gap between tested techniques and tested procedures is where many purple team programs fail. The article names the operational weakness clearly: static exercises decay as environments, tools, and detections change. That is especially relevant in financial services, where heterogeneous infrastructure and fast-moving telemetry changes make stale validation a material risk. Practitioners should build assurance around current-state evidence, not historical test completion.
Detection engineering is now a quality function. When defenders expect continuous attack simulation, detections stop being a checklist and become a product with a release and feedback cycle. That is a useful model for IAM-adjacent telemetry too, because access and endpoint controls only matter if the alerting layer keeps pace. Practitioners should manage detection content with the same discipline they apply to other production code.
AI may accelerate attack simulation, but it does not remove the need for human judgment. The article’s caution is important: AI can help generate ideas and variants, yet validation still depends on expert interpretation. That matters for agentic and machine identity programmes as well, because automated activity without interpretation can create noise rather than assurance. Practitioners should use AI to extend coverage, not to replace control ownership.
What this signals
Detection assurance is now a change-management problem. When validation is continuous, the main risk is not lack of tests but untracked environment drift. Security teams need to know which detections fail after agent updates, cloud changes, or logging path changes, because those are the moments when silent control decay begins.
Procedure granularity is the difference between coverage and evidence. If your purple team programme cannot distinguish technique labels from the specific procedure tested, it will overstate control confidence. That distinction matters for endpoint, cloud, and identity-adjacent detections where the observable behaviour is narrower than the attack family label.
For practitioners
- Automate recurring validation runs Schedule attack simulations on a fixed cadence across Windows, macOS, Linux, and cloud instances so detection gaps are surfaced as change happens, not after an annual review.
- Track procedures separately from techniques Record which procedure variant was executed for each test so a pass on a high-level ATT&CK technique does not mask incomplete behavioural coverage.
- Tie validation to detection engineering backlogs Route every failed test into the same remediation workflow you use for production detections, with owners, due dates, and retest criteria.
- Review telemetry drift after every platform change Re-run validation when EDR agents, cloud images, or logging pipelines change because the detection path can break even when the rule text has not changed.
- Start with a small technique set if resources are limited Pick five techniques that reflect your most likely threat paths and test them manually on a quarterly basis until you can automate broader coverage.
Key takeaways
- Continuous purple team programs only add value when validation is current, repeatable, and tied to how detections behave after change.
- Technique-level reporting can hide partial coverage unless teams track the specific procedures they actually exercised.
- Detection engineering works best when failed simulations feed a production-style remediation loop with retesting built in.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK addresses the attack and risk surface, while NIST CSF 2.0, CIS Controls v8 and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| MITRE ATT&CK | TA0006; TA0008 — Credential Access; Lateral Movement | The article centres on ATT&CK technique and procedure validation across real environments. |
| Recommendation — Map purple team tests to ATT&CK tactics and retest detections after each environment change. | ||
| NIST CSF 2.0 | DE.CM-01 — Monitoring for Anomalies and Events | Continuous validation supports ongoing security monitoring and detection assurance. |
| Recommendation — Use DE.CM-01 to verify that monitoring still detects the behaviors your controls are supposed to catch. | ||
| CIS Controls v8 | CIS-8 — Audit Log Management | The program depends on reliable telemetry and evidence from logging and detection pipelines. |
| Recommendation — Strengthen audit log management so validation runs can be measured against trustworthy telemetry. | ||
| NIST SP 800-53 Rev 5 | SI-4 — System Monitoring | Repeated attack simulation exists to test whether monitoring and alerting remain effective. |
| Recommendation — Apply SI-4 to confirm your monitoring stack still raises alerts after infrastructure and agent changes. | ||
Key terms
- Purple Teaming Loop: A continuous cycle that connects testing, detection, and remediation so that security findings feed directly into enforceable controls. For AI agents, the loop matters because the system’s behaviour can change after each model, prompt, or integration update.
- Procedure: A procedure is the specific way an adversary carries out a technique in a real environment. Two attackers may use the same technique but follow different procedures based on tooling, access, or operational habits. Procedure-level analysis helps defenders build more precise detections than technique labels alone.
- Detection Engineering: The discipline of designing, testing, and maintaining detection logic so it remains useful against real attacker behaviour. It covers telemetry selection, rule quality, false-positive management, and the operational workflow needed to keep alerts actionable.
- Coverage Theatre: Coverage theatre is the false confidence created when a security program reports broad test completion without proving that the most relevant behaviours were actually validated. It usually appears when teams measure activity, such as completed tests, instead of current detection effectiveness.
Deepen your knowledge
NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore the course if your programme needs a stronger governance model for credentials, workload identity, and access assurance.
Published by the NHIMG editorial team on August 18, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org