Notifications
Clear all
Topic starter
02/09/2026 1:29 pm
TL;DR: BlueBench-Simulation-002 shows that simulated Windows identity-tier attacks still expose major detection and incident-response weaknesses, with the hardest password-spray hunt averaging 33.5% and the strongest overall model reaching 77.6%, according to Cotool. The result is a reminder that directory telemetry can be highly revealing, but only when teams separate malicious identity activity from authorized administration.
NHIMG editorial — based on content published by Cotool: BlueBench-Simulation-002 on simulated Windows identity-tier attacks
By the numbers:
- Grok 4.6 led the benchmark at 77.6% accuracy, six points ahead of Claude Opus 4.8 at 71.5%.
- Four simulated Windows identity-tier attacks were scored across 14 models in 168 runs.
Questions worth separating out
Q: Why do password-spray attacks often look like normal directory activity at first?
A: Password spray can resemble routine identity noise because lockouts, password rotations, and approved bulk operations all create bursts of authentication activity.
Q: What breaks when service identities are not tracked as a distinct control group?
A: Detection logic loses the ability to separate normal service authentication from credential theft.
Q: How do teams know whether a directory change is persistence or approved administration?
A: They need lifecycle evidence tied to the change, not just the change record itself.
Practitioner guidance
- Correlate authentication bursts with lifecycle events Link password-spray alerts to password resets, bulk changes, and approved admin actions before escalating.
- Inventory service identities and owners Create a current catalogue of service accounts, their owners, and their expected authentication patterns.
- Require change provenance for directory modifications Record the business or operational reason for every privileged identity change so investigators can verify whether a change was authorised or part of a persistence chain.
What's in the full report
Cotool's full analysis covers the experimental setup and scoring detail this post intentionally leaves for the source:
- Per-task scoring breakdowns across the four simulated identity attacks, including where each model gained or lost points.
- Model-by-model cost and latency comparisons for the 168 total runs, useful if you are evaluating operational trade-offs.
- Methodology notes on how the hidden ground truth, red herrings, and deterministic detection scoring were constructed.
- Task design details for the password spray hunt, domain persistence investigation, service-credential theft detection, and hands-on-keyboard case.
👉 Read Cotool's full analysis of BlueBench-Simulation-002 identity-tier attacks →
Windows identity-tier attacks: what they reveal about detection gaps?
Explore further
02/09/2026 1:30 pm
Identity-tier attacks expose a governance blind spot, not just a detection gap. The benchmark shows that weak alerts on the directory edge are easy to misread when approved administration, lockouts, and service authentication generate similar signals. That means the real weakness is not only model performance but the absence of identity context that proves whether an action belongs to a normal lifecycle event. Practitioners should treat directory governance evidence as part of detection design, not a separate process.
A question worth separating out:
Q: Should security teams prioritise service-account visibility or broader detection tuning first?
A: Service-account visibility should come first when the main risk is credential theft or ambiguous authentication traffic. You cannot tune detections well if you do not know which identities are non-human, who owns them, and what normal looks like. Once that baseline exists, broader detection tuning becomes faster and less noisy.
👉 Read our full editorial: AI identity attacks are testing Windows control assumptions
