How do you know if red team and blue team exercises are actually improving resilience?

Why This Matters for Security Teams

Red team and blue team exercises should be judged by operational change, not by how dramatic the scenario felt. The point is to shorten the path from initial compromise to containment, improve decision quality under pressure, and expose whether controls actually hold up against realistic attacker behaviour. NIST’s control catalogue, including NIST SP 800-53 Rev 5 Security and Privacy Controls, is useful here because it maps exercises back to measurable safeguards rather than theatre.

Many organisations mistake activity for improvement. A successful exercise is not one that produces the longest report or the most findings; it is one that changes how quickly telemetry is interpreted, how confidently analysts can validate an intrusion path, and how early containment actions can be taken. If the exercise does not alter playbooks, detections, access restrictions, or escalation triggers, the security posture has probably not improved in a meaningful way.

In practice, many security teams encounter this only after a real incident exposes the same gaps that exercise debriefs had already identified.

How It Works in Practice

Improvement is usually measured by comparing repeated exercises over time against a stable set of attack objectives. Security leaders should look for reduced mean time to detect, reduced mean time to investigate, and reduced mean time to contain, but those numbers only matter when the scenarios are comparable. A change from “we saw it eventually” to “we blocked it at the perimeter” is only meaningful if the attacker path, logging coverage, and response scope were similar enough to compare.

Good measurement combines technical and operational evidence. That often includes detection coverage, alert fidelity, time to triage, containment decision time, and whether the exercised path crossed identity, endpoint, cloud, or email controls before being stopped. A mature program also checks whether the blue team used the exercise to improve detections, refine case management, and remove ambiguity from escalation steps.

• Track repeated scenarios by tactic and objective, not just by date.
• Measure whether the same path is stopped earlier on the second or third run.
• Verify that detections produce actionable context, not just noisy alerts.
• Confirm that containment steps can be executed within existing authority boundaries.
• Review whether lessons learned became control changes, not only meeting notes.

For attack-pattern mapping, MITRE ATT&CK remains a practical way to link exercise observations to adversary techniques, while the CISA Known Exploited Vulnerabilities Catalog can help ensure scenarios reflect genuinely abused weaknesses rather than hypothetical ones. Where organisations operate formal detection engineering, the exercise should also show whether SIEM rules, EDR actions, and SOAR playbooks actually triggered in time to matter.

These controls tend to break down when exercises are too bespoke, too widely announced, or too disconnected from production telemetry because the team rehearses the drill rather than testing the real response chain.

Common Variations and Edge Cases

Tighter exercise design often increases coordination overhead, requiring organisations to balance realism against the risk of disrupting business operations. That tradeoff is especially visible when production systems, regulated environments, or shared services are involved. Best practice is evolving, but there is no universal standard for how much surprise an exercise should include, and the answer usually depends on safety constraints, legal authority, and the maturity of the monitoring stack.

Some exercises improve resilience only after several cycles because the first run exposes basic visibility gaps, while later runs test response quality. Others appear successful but only because the blue team already expected the exact scenario. That is why current guidance suggests separating “did the team notice it?” from “did the team stop it?” and from “did the organisation remove the underlying weakness?”

Edge cases matter. In cloud-heavy estates, improvements may show up in identity and configuration controls before they appear in endpoint metrics. In highly segmented networks, containment may be fast but investigation still slow if log correlation is weak. In outsourced SOC models, the key question is whether exercise findings changed escalation paths and access to evidence, not just ticket closure speed. For control mapping and resilience planning, the broader NIST Cybersecurity Framework helps anchor the exercise to governance, detection, response, and recovery rather than to a single team’s performance.

Where exercise outcomes are improved on paper but not in operations, the usual cause is that findings were tracked as lessons learned instead of being converted into monitored control changes, response authority, and retested detections.

Related resources from NHI Mgmt Group

• How do you know if identity visibility is actually improving security?
• How do organisations know whether DSPM is actually improving resilience?
• How do you know if behavioural analytics are actually improving access security?
• How do you know if crisis tabletop exercises are actually working?