macOS infostealer benchmarks show IR still lags hunting

At a glance

What this is: This benchmark evaluates real macOS intrusion investigations and shows that incident response, threat hunting, and detection engineering produce very different results even when models see the same data.

Why it matters: Security teams that rely on automation for triage or hunting need to understand where workflow-specific blind spots still appear, especially when credential theft and persistence are involved.

By the numbers:

• The dataset contains 416K+ real events across 14 log sources, giving each model a broad but operationally grounded investigation surface.
• GPT-5.4 led at 86.8% accuracy overall, while Claude Opus 4.6 followed at 84.2% in the same benchmark.
• Gemini 3.1 Pro reached 89% on threat hunting but dropped to 57-58% on other tracks, showing uneven performance across investigation types.

👉 Read Cotool's analysis of the BlueBench-Intrusion-001 macOS intrusion benchmark

Context

MacOS infostealer investigations expose a common security gap: organisations often optimise for alert volume or model accuracy, but real intrusions require different reasoning across triage, hunt, and detection authoring. In this benchmark, Cotool tested a genuine Odyssey Stealer compromise in a controlled environment, which makes the primary finding relevant to blue-team operations rather than synthetic puzzle-solving.

The intrusion pattern is familiar to identity and endpoint teams. A trojanised application harvested credentials, exfiltrated browser and keychain data, and established persistence, which means the investigation touches credential governance, endpoint control, and post-compromise access review. That combination is typical of modern infostealer cases, not an edge condition.

Key questions

Q: What breaks when a macOS infostealer reaches browser and keychain data?

A: The main failure is assuming endpoint cleanup alone restores trust. Once browser and keychain secrets are stolen, the attacker may already have reusable access to other systems, so the response must include revocation, session invalidation, and targeted access review. If those steps are delayed, the malware can be removed while the identity exposure continues elsewhere.

Q: Why do stolen endpoint credentials create broader identity risk?

A: Stolen local credentials are often more valuable than the compromised host because they can authenticate to cloud services, SaaS platforms, or internal tools. That means a macOS infostealer can turn endpoint compromise into identity compromise, especially when tokens, browser sessions, or cached secrets remain valid after detection. Identity teams should treat host compromise as an account-risk event.

Q: How do you know if detection engineering is actually catching infostealer activity?

A: You know it is working when a test case can be reconstructed from telemetry as a sequence, not just as an alert. A useful detection links initial execution, credential access, and follow-on privilege use in a short window. If the logic only finds one event type, it is probably too weak to stop real infostealer tradecraft.

Q: What should teams do after a macOS infostealer incident?

A: Contain the endpoint, but do not stop there. Rotate exposed credentials, revoke active sessions, and review accounts that touched the infected device for unusual access paths. If persistence was established, verify removal of the LaunchDaemon and recheck for re-entry before returning the host to service.

Technical breakdown

How macos infostealers gain initial foothold through trojanized apps

Odyssey Stealer follows a delivery pattern that remains effective because users trust familiar software names more than the provenance of the package. The trojanized application acts as the entry point, then executes payload logic that collects browser, keychain, and other local secrets. On macOS, this often blends into normal application activity unless controls inspect notarisation, installation lineage, and execution behaviour together. The technical lesson is that application trust and user trust are not the same thing.

Practical implication: validate app provenance and block unsigned or suspicious installer paths before credential collection begins.

Credential harvesting and persistence on macOS endpoints

Once on host, infostealers target locally cached secrets because they offer immediate reuse value without needing password cracking. In this case the malware harvested browser and keychain data, then installed LaunchDaemon persistence so it could survive reboots. LaunchDaemons are effective for attackers because they run at startup with system-level reliability. That persistence layer turns a one-time infection into a repeatable credential access problem, which is why endpoint telemetry and identity signals must be correlated.

Practical implication: monitor LaunchDaemon creation and local secret access together so stolen credentials are not treated as separate events.

Why detection engineering must test short attack chains, not just alerts

The benchmark’s strongest signal is that detection engineering is a different task from hunting or incident response. A good correlation query must connect local credential validation to a subsequent privileged action within a short window, which is a behavioural pattern rather than a single indicator. That matters because infostealers often move quickly from access to use. The control problem is not whether the endpoint is noisy, but whether defenders can express a sequence-based hypothesis that survives real-world variance.

Practical implication: build detections around chained behaviours, then validate them against real multi-source telemetry rather than isolated alerts.

Threat narrative

Attacker objective: The objective is to collect reusable credentials and local secrets, then maintain persistence long enough to enable broader account compromise and data theft.

1. Entry occurred through a trojanized macOS application disguised as Ledger Live, which delivered Odyssey Stealer onto the endpoint.
2. Credential harvesting followed when the malware extracted browser and keychain data, creating immediate access to reusable secrets.
3. Persistence was established through a LaunchDaemon, allowing the actor to retain access after reboot and continue exfiltration over HTTP.
4. Impact was credential theft and exposure of local sensitive data that can be reused for account compromise and follow-on intrusion.

NHI Mgmt Group analysis

macOS infostealers are an identity problem as much as an endpoint problem: once browser and keychain secrets are stolen, the attacker has moved from malware execution into identity abuse. That shift matters because the real security failure is not only endpoint compromise, but the reuse of captured credentials against other systems. Practitioners should treat endpoint telemetry and identity telemetry as one investigation surface, not two separate programmes.

Investigation tooling must be judged by task, not by aggregate score alone: this benchmark shows that a model can look strong overall while still failing specific incident-response or detection-engineering work. That is a governance problem for blue teams because programme owners may overtrust a single headline number. The right question is which investigation phase the tool can support, and where human review still has to close the gap.

LaunchDaemon persistence is a named concept worth tracking because it converts theft into repeatable access: attackers who can survive reboot do not need to re-infect the host before each action. That creates a persistence window that increases dwell time and complicates eradication. Teams should align endpoint hardening, teardown validation, and secret revocation to the same incident timeline.

Sequence-based detection is the dividing line between hunting and alerting: the benchmark’s detection task required correlating local credential validation with a later privileged action, which is closer to attacker behaviour than to signature matching. This is the kind of control expression that fits MITRE ATT&CK-style reasoning and NIST CSF detect and respond functions. Practitioners should build detections around attack sequences, not isolated telemetry artifacts.

Identity recovery must be part of endpoint containment: when an infostealer reaches browser and keychain data, endpoint cleanup alone is not enough. The compromised secrets can outlive the malware and continue to work elsewhere. The practical conclusion is that containment should include credential revocation, session invalidation, and access review for any account exposed on the affected host.

What this signals

LaunchDaemon persistence shows why endpoint security and identity governance now overlap in practice: when a host-level compromise yields browser and keychain secrets, the next control failure is usually token reuse, not malware execution. That makes revocation speed, session invalidation, and post-exposure access review core parts of the response, not afterthoughts. The broader programme signal is that identity recovery has to be built into endpoint containment.

Model benchmarking for blue-team work is moving toward task-specific evaluation: general accuracy says too little about whether an investigation workflow can reconstruct an intrusion, hunt for evidence, or author a detection. Security leaders should expect more testing that separates incident response from threat hunting and detection engineering, because each one exercises a different control layer. For teams, the signal is to validate tools against the real work you need them to do.

Infostealer campaigns remain an access-control problem disguised as malware: stolen credentials only matter when they still work, which is why short-lived secrets, fast revocation, and exposed-session cleanup are becoming operational priorities. The practical shift is to measure how quickly identity systems can invalidate compromised access after endpoint compromise, not just how quickly the host can be cleaned.

For practitioners

• Harden macOS application trust checks Block unsigned, notarisation-bypassing, or suspiciously repackaged installers, and treat common software names as untrusted until provenance is verified. Focus monitoring on installation lineage and execution context rather than filename alone.
• Correlate local secret access with identity telemetry Alert when browser, keychain, or credential-store access is followed by authentication reuse or privilege-bearing activity in a short sequence. That correlation helps distinguish theft from routine endpoint noise.
• Track LaunchDaemon creation as a persistence indicator Investigate new LaunchDaemons immediately, especially when they appear near credential access or unusual HTTP exfiltration. Tie the artifact back to the original execution path before closing the case.
• Revocation and session invalidation should follow any secret exposure Assume browser and keychain theft exposes accounts beyond the infected host. Revoke tokens, reset affected credentials, and invalidate active sessions before the persistence mechanism is fully removed.
• Validate detections against real multi-source telemetry Use endpoint, Unified Log, and network metadata together when testing correlation logic, then measure whether the query catches short attack chains rather than isolated indicators. This is where detection engineering proves operational value.

Key takeaways

• The benchmark shows that real macOS intrusions test more than malware detection, because stolen secrets and persistence turn endpoint compromise into identity risk.
• Task-specific performance matters more than a single headline score, since incident response, hunting, and detection engineering each exposed different strengths and weaknesses.
• Containment must include revocation, session invalidation, and persistence removal, or the compromise can continue even after the endpoint is cleaned.

Key terms

• LaunchDaemon Persistence: A macOS persistence method that registers a daemon to run automatically at startup or when conditions are met. Attackers use it to keep malware active after reboot, which makes removal harder because the compromise survives a simple process kill or logout.
• Infostealer: An infostealer is malware built to collect credentials, session material, tokens, and other authentication data from infected systems. In NHI programmes, the risk is not only theft but reuse, because harvested workload secrets can unlock cloud access long after the initial infection.
• Sequence-based detection: An analytics approach that evaluates the order of identity events rather than each event alone. For NHI and SaaS security, it is essential because login anomalies, factor enrollment, and bulk downloads become far more meaningful when they occur in a suspicious sequence.

What's in the full report

Cotool's full analysis covers the operational detail this post intentionally leaves for the source:

• The per-track question set and scoring methodology used to compare incident response, threat hunting, and detection engineering performance.
• The full SQL-based investigation flow that shows how the benchmark validated answers against live telemetry.
• The task-by-task model recommendations, including where each model performed best and where it struggled.
• The complete dataset context for the real macOS intrusion and the 14 log sources used in the sandbox.

👉 The full Cotool benchmark details the task design, scoring method, and model-by-model performance.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle control. It gives identity and security practitioners a structured way to connect access risk to operational controls.