Notifications
Clear all
Topic starter
02/08/2026 9:38 am
TL;DR: ASW-Bench tested frontier model CLIs against realistic SOC tasks and found that all models identified the initial C2 beacon, but only Claude consistently traced credential theft, lateral movement, and event log clearing, according to LimaCharlie. The result suggests agentic SecOps still depends on strong workflow design, not model capability alone.
NHIMG editorial — based on content published by LimaCharlie: What Frontier Models Can Actually Do in a SOC
Questions worth separating out
Q: What breaks when AI only detects the first stage of a SOC incident?
A: The investigation stops before the compromise is understood.
Q: Why do AI-assisted SOC tools still depend on good identity telemetry?
A: Because attack reconstruction often turns on who accessed what, when, and from where.
Q: How can organisations tell whether agentic SecOps is under control?
A: Look for evidence that the workflow is bounded, reviewable, and reversible.
Practitioner guidance
- Benchmark AI on multi-stage investigations Use scenarios that require the model to move from beaconing to credential theft, lateral movement, persistence, and exfiltration.
- Require evidence-linked output for incident summaries Do not accept a narrative unless it cites the telemetry or event types that support each conclusion.
- Validate telemetry coverage before AI rollout Check whether the SOC platform exposes the logs, process data, and identity context needed for the model to investigate credential abuse and privilege movement.
What's in the full article
LimaCharlie's full blog covers the operational detail this post intentionally leaves for the source:
- The ASW-Bench scenario design and the exact prompt structure used to test model behaviour in a SOC workflow.
- Per-model raw outputs and baseline logs showing how Claude, Codex, and Gemini differed across investigation stages.
- The open-source benchmark framework and scenario access for teams that want to reproduce the test internally.
- The underlying reasoning for choosing LimaCharlie as the tool environment rather than a stitched-together open-source stack.
👉 Read LimaCharlie's analysis of frontier model performance in SOC investigations →
Frontier models in SOC work: what do the benchmark results show?
Explore further
02/08/2026 12:33 pm
AI-assisted SOC is now a control problem, not a productivity feature. The benchmark shows that frontier models can identify the obvious start of an attack, but that does not mean they can sustain the investigative chain needed for response. For identity teams, this matters because SOC decisions often hinge on whether credential abuse, privilege movement, or persistence are fully traced. A model that stops at the beacon leaves the control gap open.
A question worth separating out:
Q: When should organisations keep humans in control of AI-driven investigations?
A: Keep humans in control whenever the model is being used to justify containment, escalation, or case closure. If the system cannot reliably explain intermediate stages such as persistence or exfiltration, it should support the analyst rather than decide for the analyst. Trust should follow proven fidelity, not interface convenience.
👉 Read our full editorial: Frontier models in SOC work still miss key attack stages
