macOS infostealer benchmarks show IR still lags hunting

At a glance

What this is: This benchmark evaluates nine models against a real macOS Odyssey Stealer intrusion and finds that performance differs materially by investigation task, with incident response proving hardest.

Why it matters: For IAM, NHI, and SOC practitioners, the benchmark is a reminder that detection and response workflows need evidence-specific controls, not just broad model capability.

By the numbers:

• The benchmark spans 36 tasks across 9 models and 416K+ log events.
• Incident response, threat hunting, and detection engineering each account for 12 tasks, or 33%, of the benchmark.
• GPT-5.4 led the benchmark with 86.8% accuracy, while Gemini 3.0 Flash scored 54.0%.
• Gemini 3.1 Pro reached 89% on threat hunting but fell to 57-58% on other tracks.

Context

macOS infostealer investigations are a useful stress test because they combine endpoint telemetry, browser and keychain evidence, and attacker persistence artefacts into one incident workflow. In this benchmark, the primary issue is not whether a model can answer security questions in the abstract, but whether it can follow a real intrusion across logs and preserve investigative context.

For identity and access teams, the relevance is direct. Infostealers target credentials, browser sessions, and local trust material, which makes the boundary between endpoint compromise and identity compromise much thinner than many programmes assume. That is why response quality matters as much as detection quality when stolen access can become the next control failure.

The starting point is typical of modern infostealer cases rather than exceptional: one compromised workstation can become a credential loss event, an investigation event, and a detection-engineering test at the same time.

Key questions

Q: What breaks when a macOS infostealer reaches browser and keychain data?

A: The main failure is assuming endpoint cleanup alone restores trust. Once browser and keychain secrets are stolen, the attacker may already have reusable access to other systems, so the response must include revocation, session invalidation, and targeted access review. If those steps are delayed, the malware can be removed while the identity exposure continues elsewhere.

Q: Why does incident response depend so heavily on identity governance?

A: Because most incidents move through identities, entitlements, and privileged paths before they are fully understood. Identity governance determines whether teams can tell which account was used, which systems were exposed, and who can shut access down. Without that visibility and authority, containment becomes slower and recovery becomes less trustworthy.

Q: What do security teams get wrong about investigation benchmarks?

A: They often treat benchmark accuracy as a single score, but real investigations reward different skills in incident response, threat hunting, and detection engineering. A model can perform well in one phase and still miss the context needed to guide containment or create a durable detection.

Q: How should teams use real intrusion data to improve detections?

A: Use it to test whether a correlation can survive the full attack sequence, from delivery through persistence and exfiltration. If a rule only works on isolated alerts, it is not ready for operational use against a real compromise.

Technical breakdown

How a trojanised macOS app becomes an identity event

The benchmark centres on Odyssey Stealer, which was delivered through a trojanised app that impersonated a legitimate Ledger Live installer. That delivery pattern matters because the initial compromise is not just malware execution, it is a trust breach at the point where users grant the application legitimacy. Once executed, the stealer harvested browser data and keychain material, turning endpoint access into credential exposure. For identity teams, this is a classic example of endpoint compromise cascading into authentication compromise, because the attacker is no longer limited to the device and can pivot into accounts and sessions that trust the stolen material.

Practical implication: treat malicious app delivery as an identity incident, not only an endpoint incident.

Why incident response is harder than hunting in real intrusion data

The dataset shows that different investigative tasks reward different reasoning paths. Incident response requires reconstruction of sequence, artefact location, and containment relevance, while threat hunting can often rely on pattern recognition and hypothesis testing over a narrower slice of telemetry. Detection engineering adds another layer because the analyst must express the intrusion as a durable query or correlation rule. In practice, IR is hardest when evidence is distributed across logs, network metadata, and local host artefacts, because the investigator has to preserve chain-of-events context rather than simply identify suspicious behaviour.

Practical implication: validate IR workflows against full attack narratives, not only alert triage snippets.

Why persistence and exfiltration clues matter for credential governance

Odyssey Stealer installs LaunchDaemon persistence and exfiltrates keychain and browser data over HTTP, which means the compromise has both durable presence and immediate data movement. Persistence gives the operator a way to return after the first execution, while exfiltration turns cached credentials and session material into usable access. For NHI and IAM practitioners, the governance lesson is that secrets, browser tokens, and local trust artefacts behave like high-value credentials once exposed. That is why account hygiene, token scope, and revocation speed need to align with endpoint detection, not sit in separate operational silos.

Practical implication: couple credential revocation and token invalidation to endpoint compromise handling.

Threat narrative

Attacker objective: The attacker aims to capture usable credentials and session material from macOS endpoints while maintaining persistence for continued access.

1. Entry occurred when a trojanised fake Ledger Live application delivered the Odyssey Stealer payload onto a macOS endpoint.
2. Credential theft followed as the stealer harvested browser data and keychain material from the compromised device.
3. Persistence was established through a LaunchDaemon, allowing the operator to maintain access beyond the initial execution.
4. Impact came from exfiltration of stolen identity material over HTTP, creating downstream account compromise risk.

Breaches seen in the wild

• Nx s1ngularity attack 2025: Attackers stole Nx's npm token via a GitHub Actions flaw and shipped malware that stole 2,349 secrets and abused developers' AI CLIs.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Incident response for infostealers is an identity problem before it is a malware problem. Once browser sessions, keychain material, or saved credentials are harvested, the attacker has moved from endpoint execution into account risk. That changes the governance question from binary infection status to trust-material exposure and revocation scope. Practitioners should treat this as a credential lifecycle failure, not just a host compromise.

Investigation quality depends on context preservation, not model generality. The benchmark shows that a model can be strong at threat hunting and still underperform when asked to reconstruct a compromise end to end. That gap matters because real responders need ordered evidence, not isolated indicators. The named concept here is investigative context loss: when tooling breaks the chain between host events, network traces, and identity artefacts, response quality degrades quickly. Practitioners should design workflows that retain chronology across telemetry layers.

MacOS infostealers expose the weak boundary between endpoint security and IAM. Browser tokens, keychain entries, and cached authentication data are effectively identity assets once stolen. That means revocation, session invalidation, and detection engineering have to be treated as one control plane. NHI Mgmt Group’s position is that many programmes still separate endpoint response from identity response, and this benchmark shows why that separation is operationally unsafe.

Detection engineering should be judged by whether it can turn intrusion evidence into durable queries. The benchmark’s third track rewards the ability to build and validate correlations from real compromise data, not just to describe the malware. That aligns with mature SOC practice: the control is not only finding an event, but codifying it into repeatable detection logic. Practitioners should use real intrusions to test whether their detections survive noisy, multi-source evidence.

AI-assisted investigation tools need task-specific guardrails when the evidence path crosses identity data. The benchmark demonstrates that one investigation mode does not represent all operational needs. When a tool is asked to reason over credentials, persistence, and exfiltration at once, the risk is not only false confidence but also incomplete triage. Practitioners should scope AI use to the stage of the investigation it actually supports and require human review where identity compromise is in play.

What this signals

Investigative context loss: the practical failure mode in this benchmark is not raw model weakness, but loss of chain-of-events context across telemetry layers. Security programmes should treat endpoint, network, and identity evidence as a single investigation fabric, because infostealers move from device compromise to account risk in one sequence.

The benchmark also reinforces a familiar governance problem: response workflows are often optimised for alert handling, while real intrusions demand artefact correlation, chronology, and decision traceability. If an AI-assisted workflow cannot preserve those three things, it should be constrained to narrow triage tasks rather than full-case analysis.

For practitioners

• Map infostealer cases to identity incident handling Classify browser-token and keychain theft as a credential exposure event so revocation, session invalidation, and account review start immediately after host compromise is confirmed.
• Correlate endpoint, network, and identity evidence Build investigation workflows that preserve chronology across EDR telemetry, macOS logs, and network metadata so analysts can reconstruct the full attack path instead of isolated alerts.
• Validate detections against real intrusion sequences Test correlation rules on full compromise chains, including delivery, persistence, and exfiltration, to verify that detections survive multi-source evidence and not just clean lab patterns.
• Harden revocation for stolen browser and keychain material Shorten the window between suspected compromise and token or password revocation, because infostealers turn cached identity material into usable access very quickly.

Key takeaways

• Real macOS infostealer incidents collapse endpoint security and identity security into the same operational problem because stolen browser and keychain data can become immediate account access.
• Benchmark results vary sharply by task, which shows that incident response, threat hunting, and detection engineering require different forms of evidence handling and reasoning.
• The strongest control response is faster credential revocation, tighter session invalidation, and investigation workflows that preserve the full compromise chain across telemetry sources.

Key terms

• Infostealer: An infostealer is malware built to collect credentials, session material, tokens, and other authentication data from infected systems. In NHI programmes, the risk is not only theft but reuse, because harvested workload secrets can unlock cloud access long after the initial infection.
• LaunchDaemon Persistence: A macOS persistence method that registers a daemon to run automatically at startup or when conditions are met. Attackers use it to keep malware active after reboot, which makes removal harder because the compromise survives a simple process kill or logout.
• Credential exfiltration: Credential exfiltration is the theft of usable authentication material such as tokens, keys, or certificates. In NHI environments, the stolen item is often already valid and can be replayed immediately. That is why detection must be paired with revocation and entitlement review rather than relying on alerts alone.
• Correlated Investigation: A correlated investigation is a case-building approach that joins identity, activity and context across tools into one narrative. Instead of judging each alert separately, the investigator reconstructs sequence and intent from multiple systems so legitimate access, misuse and exfiltration can be distinguished more reliably.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps identity and security practitioners connect compromise handling to the credential lifecycle.