Frontier models in SOC work still miss key attack stages

At a glance

What this is: This benchmark tests what frontier models can actually do in SOC investigations, and finds that strong detection of obvious signals does not translate into full attack-chain analysis.

Why it matters: It matters because IAM, PAM, and NHI teams are increasingly considering AI-assisted operations, and incomplete investigation capability can leave credential abuse, lateral movement, and persistence insufficiently contained.

👉 Read LimaCharlie's analysis of frontier model performance in SOC investigations

Context

Agentic SecOps promises to help analysts triage alerts, reconstruct attack chains, and draft incident reports, but that only matters if the model can follow a real workflow rather than answer abstract questions. This benchmark focuses on the operational gap between cybersecurity knowledge and usable security work, which is where AI-assisted SOC tooling often fails in practice.

The identity angle is indirect but real: once AI tools are granted access to logs, consoles, and investigation workflows, they become part of the control plane that handles credentials, privileges, and incident evidence. That means their limitations affect how well teams can detect abuse of NHIs, service accounts, and standing access during an active attack.

The starting position here is typical of the market, not unusual. Many teams are testing model capability before they have defined the guardrails, validation criteria, and audit expectations needed for production use.

Key questions

Q: What breaks when AI only detects the first stage of a SOC incident?

A: The investigation stops before the compromise is understood. If a model can identify a beacon but cannot trace credential theft, lateral movement, or persistence, analysts may underestimate blast radius and miss the controls that actually failed. That makes the output useful for triage, but unsafe as a standalone incident narrative or response record.

Q: Why do AI-assisted SOC tools still depend on good identity telemetry?

A: Because attack reconstruction often turns on who accessed what, when, and from where. Without identity and privilege context, a model may recognise malicious activity but fail to connect it to the right account, service principal, or lateral path. Identity telemetry turns raw alerts into actionable investigation evidence.

Q: How can organisations tell whether agentic SecOps is under control?

A: Look for evidence that the workflow is bounded, reviewable, and reversible. You should be able to show who approved each action, what data the agent touched, which tenant or environment it operated in, and whether it can be stopped without breaking the rest of the response process. If you cannot trace those points, control is incomplete.

Q: When should organisations keep humans in control of AI-driven investigations?

A: Keep humans in control whenever the model is being used to justify containment, escalation, or case closure. If the system cannot reliably explain intermediate stages such as persistence or exfiltration, it should support the analyst rather than decide for the analyst. Trust should follow proven fidelity, not interface convenience.

Technical breakdown

Why benchmark agentic SecOps against real attack chains

A useful benchmark for SOC automation has to measure more than cybersecurity knowledge. It needs to show whether a model can take an alert, preserve context, correlate evidence, and narrate an attack chain with enough fidelity to support analyst action. That is different from answering exam-style questions. In this case, the scenario included beaconing, credential theft, lateral movement, persistence, defense evasion, and DNS exfiltration, which are the kinds of stages analysts actually need to connect. Models that only identify the first visible signal can still leave the rest of the compromise uninvestigated.

Practical implication: benchmark AI on end-to-end investigative workflows, not isolated detections.

How CLI-based tool access changes model behaviour in the SOC

Giving a model access to a security platform CLI creates a constrained agentic loop: the model can query, inspect, and summarise, but it still depends on the available telemetry and the prompt structure. That setup is valuable because it strips away custom middleware and shows what the model can do with minimal scaffolding. The trade-off is that performance may reflect both model reasoning and the quality of the underlying telemetry. For SOC teams, this means platform integration quality can matter as much as model selection when evaluating agentic security operations.

Practical implication: test the full tool path, including telemetry quality and command design, before you trust the result.

Why incident report quality matters as much as detection

SOC work does not end at finding malicious activity. Analysts need a coherent incident narrative that maps evidence to tactics, explains scope, and supports containment and recovery decisions. The benchmark shows that some models can produce a multi-phase narrative, while others miss intermediate steps such as credential access or lateral movement. In operational terms, a model that cannot reliably describe the chain may still be useful for triage, but it cannot be treated as a source of record for response decisions or post-incident review.

Practical implication: require evidence-linked narrative output before using AI for incident summarisation.

Threat narrative

Attacker objective: The attacker aims to maintain access, move laterally, and exfiltrate data while reducing the visibility of the compromise.

1. Entry began with a realistic post-exploitation sequence that started from malicious beaconing and command-and-control communication visible in the SOC telemetry.
2. Credential access and escalation followed through LSASS and SAM/SYSTEM dumping, enabling theft of material that would support further movement and persistence.
3. Lateral movement and impact potential appeared through ARP or ping sweeps, SMB enumeration, WMI persistence, event log clearing, and DNS exfiltration attempts.

NHI Mgmt Group analysis

AI-assisted SOC is now a control problem, not a productivity feature. The benchmark shows that frontier models can identify the obvious start of an attack, but that does not mean they can sustain the investigative chain needed for response. For identity teams, this matters because SOC decisions often hinge on whether credential abuse, privilege movement, or persistence are fully traced. A model that stops at the beacon leaves the control gap open.

Human analysts still anchor trust in the investigation layer. The useful output is not a generic summary, but a defensible chain of evidence that can survive review. That raises the bar for AI in security operations: the model must support auditability, not just speed. For IAM and PAM programmes, the practical takeaway is that AI output must be treated as decision support until it can consistently explain credential access and privilege movement.

Credential visibility remains the named failure mode in agentic SecOps. The benchmark highlights a basic governance assumption that is often false: if a tool can see the alert, it can also see the identity trail behind it. In practice, partial telemetry, incomplete event coverage, and weak context stitching still break the investigation. Teams should treat credential visibility as a prerequisite for any AI-assisted investigation workflow.

Agentic SecOps will converge with identity governance. Once AI systems are allowed to investigate, they inherit the same need for access scoping, logging, and accountability that human analysts already face. That makes IAM, PAM, and NHI governance part of AI operations design, not a separate concern. The programme implication is clear: if the model can touch security tools, it needs identity controls of its own.

ASW-Bench exposes the detection-response latency gap. The benchmark suggests that many environments can surface an alert faster than they can produce a complete, trustworthy narrative about what happened next. That gap is where adversaries benefit, especially when persistence and exfiltration are less visible than the initial beacon. Practitioners should measure how long it takes to move from alert to evidence-backed scope, not just alert to acknowledgement.

What this signals

Model capability is advancing faster than control assurance. The benchmark suggests that security teams will be pressured to accept AI assistance before they have defined what a trustworthy investigation output looks like. That creates a governance problem as much as a technical one, because SOC automation must now prove fidelity across identity, telemetry, and evidence handling.

Identity context will become the differentiator in AI-assisted operations. If a model cannot connect alerts to accounts, service identities, and privilege paths, it cannot reliably assess blast radius. Teams should treat identity telemetry as a core dependency for agentic SecOps, not an optional enrichment layer.

The practical question is whether your programme can audit AI-driven decisions with the same discipline applied to human analysts. That means measuring investigation completeness, evidence traceability, and handoff points, then comparing them with platform access scope and logging quality.

For practitioners

• Benchmark AI on multi-stage investigations Use scenarios that require the model to move from beaconing to credential theft, lateral movement, persistence, and exfiltration. Measure whether it can explain each stage with evidence rather than only naming the first alert hit.
• Require evidence-linked output for incident summaries Do not accept a narrative unless it cites the telemetry or event types that support each conclusion. This reduces the risk of AI-generated confidence that is not grounded in the investigation record.
• Validate telemetry coverage before AI rollout Check whether the SOC platform exposes the logs, process data, and identity context needed for the model to investigate credential abuse and privilege movement. Missing telemetry will look like model failure, but the real issue may be observation gaps.
• Separate triage assistance from response authority Let models accelerate first-pass analysis, but keep containment, escalation, and closure decisions with human analysts until the system proves it can reliably reconstruct the attack chain.

Key takeaways

• Frontier models can recognise the start of an attack, but that does not mean they can complete a defensible SOC investigation.
• The benchmark shows that credential theft, lateral movement, and exfiltration remain the hard parts, which is exactly where investigation quality matters most.
• Security teams should treat AI in the SOC as a controlled workflow dependency, with identity telemetry and evidence validation built in from the start.

Key terms

• Agentic SecOps: A security operations model in which AI systems can coordinate tasks and take bounded actions across alert handling, investigation, and response. The critical question is whether those actions are governed, explainable, and reversible enough to fit enterprise control requirements.
• Attack Chain Reconstruction: The act of linking events across time and controls to explain how an attacker moved from entry to impact. For modern fraud, reconstruction is essential because the meaningful threat often appears only after automated and manual stages are combined.
• Detection-Response Latency: The elapsed time between identifying a security issue and executing a bounded, auditable fix. In data security programmes, long latency means exposure persists after discovery, which undermines the value of detection and weakens compliance evidence.

What's in the full article

LimaCharlie's full blog covers the operational detail this post intentionally leaves for the source:

• The ASW-Bench scenario design and the exact prompt structure used to test model behaviour in a SOC workflow.
• Per-model raw outputs and baseline logs showing how Claude, Codex, and Gemini differed across investigation stages.
• The open-source benchmark framework and scenario access for teams that want to reproduce the test internally.
• The underlying reasoning for choosing LimaCharlie as the tool environment rather than a stitched-together open-source stack.

👉 The full LimaCharlie post includes benchmark methodology, raw outputs, and attack-scenario details.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to connect identity controls to broader security operations and governance decisions.