Red Team Realism Gap

Expanded Definition

The red team realism gap describes the distance between an exercise that looks convincingly adversarial and one that actually mirrors how threat actors behave in production environments. In practice, the gap appears when testing emphasizes isolated exploits, scripted kill chains, or headline friendly techniques, while real intrusions rely on quieter tradecraft such as credential abuse, identity pivoting, living off the land activity, and persistence that blends into normal operations. NHI Management Group treats this as a governance problem as much as a testing problem because a report can be technically accurate yet strategically misleading if it does not reflect the organisation’s true exposure.

Definitions vary across vendors and practitioners because no single standard governs how realistic a red team must be. The most useful yardstick is whether the exercise reflects the organisation’s current threat model, business critical assets, and detection limitations, rather than whether it simply produced an entertaining intrusion path. NIST Cybersecurity Framework 2.0 is helpful here because it frames cybersecurity as an ongoing governance and risk management activity, not a one-time demonstration of technical skill. The most common misapplication is treating a polished red team success as proof of resilience when the exercise never attempted the attacker behaviours most likely to occur in the environment.

Examples and Use Cases

Implementing red teaming rigorously often introduces scheduling, access, and safety constraints, requiring organisations to weigh operational disruption against the realism needed to expose meaningful gaps.

• A team tests a public-facing web application with a known exploit, while real attackers are more likely to start with stolen credentials and abuse weak session controls.
• An exercise focuses on phishing email clicks, but the organisation’s highest risk comes from compromised service accounts and poorly governed secrets in automation pipelines.
• A red team demonstrates domain admin compromise through a noisy technique, while actual adversaries use lateral movement through trusted remote tools and legitimate administration paths.
• An internal test avoids identity systems entirely, even though NIST Cybersecurity Framework 2.0 would push the organisation to evaluate risk across the full environment, including identity and access dependencies.
• A cloud assessment validates alerting on malware execution, but skips API token theft, misused roles, and service-to-service trust, which are the more plausible attacker routes in that architecture.

In mature programs, the realism target is usually tied to the organisation’s threat intelligence, incident history, and material exposure. That means a financial services firm might prioritise account takeover and privileged access abuse, while a software company might focus on build systems, code signing, and non-human identities. Where AI systems are in scope, NIST AI 600-1 helps clarify that testing should consider how model access, prompts, tools, and downstream actions can be abused, not just whether a model can be tricked into producing unsafe text. Useful context also comes from NIST AI 600-1, especially when agentic workflows are part of the attack surface.

Why It Matters for Security Teams

The realism gap matters because defensive confidence is often built on the outputs of red team exercise. If those outputs overstate resilience, leaders may defer controls that would have reduced real attacker dwell time, limited identity compromise, or protected high-value assets. This is especially important in environments where NHI, privileged automation, and agentic AI expand the number of execution paths an attacker can abuse without touching traditional endpoints. A red team that ignores secrets sprawl, weak token lifecycle management, or over-privileged service identities can miss the very behaviours that modern intrusions exploit.

For security teams, the lesson is not to demand perfect realism, but to define what realism means for the organisation’s risk profile and then verify that the exercise actually covers it. This aligns well with the broader governance intent of NIST Cybersecurity Framework 2.0 and the AI governance emphasis in NIST AI Risk Management Framework, both of which support risk-informed testing rather than theatre. Organisations typically encounter the true cost of the realism gap only after a real intrusion bypasses the exact controls their red team never meaningfully challenged, at which point the exercise findings become operationally unavoidable to revisit.

Related resources from NHI Mgmt Group

• Should organisations require reproducible evidence from AI red-team tests?
• How do runtime guardrails differ from red team findings in AI governance?
• How should security teams red team AI agents that use tools and memory?
• Why do traditional red team exercises miss so many AI security issues?