Why do annual ATT&CK tests fall short for modern detection programmes?

Why This Matters for Security Teams

Annual ATT&CK testing gives teams a point-in-time view, but detection programmes fail in the gaps between test cycles. Modern environments change too quickly: cloud services are reconfigured, endpoints are rebuilt, identity paths shift, and telemetry pipelines are tuned or broken. A test that was valid last quarter can become misleading after a logging change or an authentication control update. The NIST Cybersecurity Framework 2.0 emphasises ongoing governance and continuous improvement, which is a better fit for detection assurance than a yearly exercise.

The real issue is not whether ATT&CK is useful, but how it is used. ATT&CK is strongest as an adversary behaviour model for validating coverage, prioritising detections, and identifying blind spots. It is weaker when treated as a once-a-year compliance artefact that proves resilience across a fast-moving estate. Security leaders often overestimate coverage because a test passed in a controlled lab, while production conditions, identity risk, and telemetry quality are different in practice. In practice, many security teams encounter detection gaps only after an incident or major platform change has already exposed them, rather than through intentional validation.

How It Works in Practice

Continuous validation works by turning detection testing into an operational control rather than a periodic project. Instead of waiting for an annual red-team or purple-team engagement, teams schedule smaller, repeatable checks that reflect current assets, current telemetry, and current identity states. That means validating whether a use case still fires after a SIEM parser changes, whether cloud logs still include the needed fields, and whether identity telemetry still supports suspicious sign-in and privilege escalation detection.

A practical programme usually combines multiple layers:

• Map high-value ATT&CK techniques to critical detection use cases.
• Re-test those use cases whenever logging, identity policy, or endpoint tooling changes.
• Use synthetic events or controlled adversary emulation to confirm alerts still trigger.
• Track both detection presence and detection quality, including fidelity, timing, and analyst usefulness.
• Review identity coverage for privileged accounts, service identities, and authentication anomalies using guidance from the NIST SP 800-63 Digital Identity Guidelines where identity assurance matters.

ATT&CK remains the behavioural taxonomy, but the operational discipline is continuous telemetry assurance. That distinction matters because coverage is not just about whether a rule exists. It is about whether the right data still reaches the right rule, whether enrichment is still accurate, and whether the alert is still actionable for the SOC. The MITRE ATT&CK Enterprise Matrix is useful here as the reference model for selecting techniques and organising tests around realistic attacker behaviour.

Teams that mature this practice often integrate validation into change management, CI/CD, and security operations reporting. That creates a feedback loop where detection gaps are found soon after configuration drift, not months later. These controls tend to break down when telemetry ownership is fragmented across cloud, endpoint, and identity teams because no single group can prove the test still reflects the live environment.

Common Variations and Edge Cases

Tighter continuous testing often increases operational overhead, requiring organisations to balance assurance against engineering capacity. That tradeoff is especially visible in distributed cloud estates, high-churn engineering environments, and identity-heavy architectures where logs, tokens, and permissions change frequently.

There is no universal standard for how often validation should run. Current guidance suggests the cadence should be driven by risk, change velocity, and detection criticality rather than an arbitrary annual date. For some teams, daily checks are realistic for a small set of crown-jewel detections. For others, weekly or event-driven validation is the better fit. The key is that the schedule should follow business and technical change, not calendar convenience.

Edge cases also matter. If a control depends on third-party SaaS logs, the team may not control schema changes or delivery delays, so validation must include data integrity checks. In identity-centric environments, a control can appear healthy even when MFA, session policy, or privileged access workflows have shifted in ways that alter detection behaviour. This is where detection testing intersects with identity governance: if access paths change, the detection model must change with them. Without that linkage, annual ATT&CK exercises can provide false reassurance, especially when a major platform migration or logging redesign has just happened.

Related resources from NHI Mgmt Group

• Why do backup and disaster recovery controls fall short for modern resilience programmes?
• Why do static role-based policies fall short in zero trust programmes?
• Why do access reviews often fall short in ISO 27001 programmes?
• Why do perimeter-based zero trust models fall short for AI programmes?