Frontier models in SOC work still miss key attack stages

At a glance

What this is: LimaCharlie’s ASW-Bench shows frontier model CLIs can identify obvious alert signals, but they still miss important post-exploitation stages in realistic SOC investigations.

Why it matters: For SOC, SIEM, and SecOps teams, the finding matters because it shows agentic AI can assist triage but cannot yet be trusted to complete an investigation without strong workflow controls and human validation.

Context

A SOC benchmark only matters if it tests investigation work, not trivia. Here the gap is between recognising an alert and reconstructing the attack chain, which is where analyst judgement, tool access, and evidence handling still determine whether automation helps or misleads.

This is also an identity and governance problem in disguise. Once an AI system can touch logs, endpoints, and response tooling, its access and decision boundaries matter as much as its detection quality, especially when it is acting like an analyst but is not governed like one.

Key questions

Q: Why do SOC AI tools fail when they can spot the beacon but not the rest of the attack chain?

A: Because alert recognition is only the first step. Effective investigation requires correlating credential theft, lateral movement, persistence, and cleanup across multiple data sources. If a model cannot reconstruct those stages, it may produce a plausible summary while missing the attacker’s real scope and containment requirements.

Q: How should security teams implement agentic AI in SOC workflows safely?

A: Start with narrow, high-confidence use cases such as alert triage and evidence gathering, then require explicit policy gates before any remediation action. Use dedicated machine identities, least privilege, and full audit logging so the AI cannot exceed its assigned scope. The safest deployments treat autonomy as a controlled exception, not the default operating mode.

Q: What are the signs that an AI SOC assistant is not ready for production use?

A: It gives coherent summaries but misses key stages such as credential abuse, lateral movement, or exfiltration. Another warning sign is when its output depends heavily on hidden prompt tuning or custom orchestration that will not exist in production. That means the apparent capability is not transferable.

Q: Should organisations treat an AI SOC assistant as an analyst or as a controlled system?

A: As a controlled system. Once it can query logs, correlate events, or propose response actions, it has an effective identity and permission boundary that must be governed. Treat it like an operating identity with scoped access, auditability, and human approval for any action that changes the environment.

Technical breakdown

Why alert recognition is easier than attack reconstruction

Most frontier models are strong at pattern matching, so a beacon, hash, or suspicious process tree is easy to label. Investigation is different. It requires joining telemetry across time, preserving sequence, and deciding which signals indicate initial access, credential abuse, persistence, or cleanup. In a SOC context, that means the model must reason across SIEM events, endpoint evidence, and response artifacts rather than answer isolated prompts. The benchmark design matters because it measures whether a model can operate inside that workflow with minimal scaffolding, not whether it can describe a threat in abstract terms.

Practical implication: judge SOC AI by multi-stage investigation performance, not by single-alert classification accuracy.

What the missed stages tell us about agentic SecOps

The missed stages are the important part. Credential theft, lateral movement, persistence, and log clearing are not just extra details, they are the stages that establish scope and confidence. If a model fails there, it may still generate a plausible report while overlooking the attacker’s real access path or containment needs. This is why agentic SecOps must be treated as a control system, not a chatbot. The model needs bounded tool use, evidence provenance, and explicit escalation points so output quality does not depend on luck or hidden prompt engineering.

Practical implication: design agentic SOC workflows so the model can assist analysis without becoming the sole source of investigative truth.

Why observability and access boundaries matter for AI analysts

A benchmark like ASW-Bench also exposes a governance issue. If an AI assistant can query security tooling, it becomes an operating identity with access, scope, and accountability requirements. That means observability, auditability, and least-privilege design are not optional extras. The same controls that govern human analysts must increasingly govern agentic systems, especially when the system is allowed to read incidents, query endpoints, or draft response actions. The operational question is no longer whether AI can help, but whether its access path is sufficiently bounded to trust the output.

Practical implication: assign AI security tools explicit identity, permission scope, and audit trails before expanding their SOC role.

Threat narrative

Attacker objective: The objective is to maintain foothold, move laterally, and exfiltrate data while evading detection and response.

1. Entry begins with a realistic post-exploitation chain that starts from an observed malicious beacon and command-and-control channel in the test scenario.
2. Credential access follows when the attack chain includes credential theft through LSASS and SAM or SYSTEM dumps, giving the attacker stronger foothold and reach.
3. Escalation and lateral movement occur through ARP or ping sweeps, SMB enumeration, and persistence behavior that expands access across the environment.
4. Impact is the attacker’s ability to hide activity, move through systems, and exfiltrate data such as DNS traffic if the chain is not fully reconstructed.

Breaches seen in the wild

• Nx s1ngularity attack 2025: Attackers stole Nx's npm token via a GitHub Actions flaw and shipped malware that stole 2,349 secrets and abused developers' AI CLIs.
• PyPI secrets exposure 2023: Researchers found 3,938 unique secrets in published PyPI packages, 768 still valid; a new release or yank does not remove them.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Benchmarking SOC AI by alert recognition alone is misleading. A model that can label a beacon has not yet demonstrated investigative competence. The hard part of SOC work is reconstructing stage progression across credential access, lateral movement, and cleanup, and that is where the benchmark shows clear gaps. Practitioners should treat single-stage detection as a weak signal, not a deployment threshold.

Agentic SecOps creates a governance problem as soon as the model can touch live tools. Once the model is querying telemetry or drafting incident narratives, it behaves like an operating identity with real privileges. That means access scope, logging, and auditability matter as much as raw accuracy. Teams that ignore the identity of the AI system will overestimate the reliability of its conclusions.

Security operations will increasingly depend on workflow design, not model hype. The benchmark suggests baseline model capability is useful but insufficient without careful orchestration, evidence chaining, and human review. That is the same pattern seen in other automation domains: performance comes from how the system is governed, not from the model alone. Practitioners should design for bounded autonomy, not open-ended analyst substitution.

Attack reconstruction is now a measurable control objective. The fact that no model found DNS exfiltration shows why output quality must be tested against complete kill-chain coverage, not just obvious initial indicators. A SOC programme that cannot verify stage coverage has a blind spot in both detection and response. Teams should make multi-stage reconstruction part of their AI acceptance criteria.

Frontier model capability is advancing faster than SOC governance models. That creates a temporary mismatch in which teams may have access to tools that can assist analysis, but not yet the assurance needed to rely on them. The right response is not to wait for perfect models, but to formalise approval gates, logging, and scoped access now. Governance must catch up before scale does.

What this signals

AI SOC capability should be measured against the attack chain, not the alert queue. If a model cannot connect beaconing to credential abuse, lateral movement, and cleanup, it is not yet dependable for incident reconstruction. Teams should set acceptance criteria around stage coverage, not just detection quality.

Agentic SecOps introduces a new kind of operating identity. The moment a model can query logs or draft response actions, access scope and auditability become governance requirements, not implementation details. That pushes SOC teams toward identity-style controls for AI systems, including scoped permissions and traceable action histories.

For practitioners

• Define benchmark criteria for SOC AI Require evaluation against complete investigation stages, including initial access, credential theft, lateral movement, persistence, cleanup, and exfiltration.
• Bound AI access to security tooling Give agentic SOC systems only the minimum permissions needed to read alerts, query logs, and draft summaries, with explicit denial of destructive actions.
• Add audit trails for model-driven investigations Log every query, retrieved event, correlation, and recommended action so analysts can reconstruct how the model reached its conclusion.
• Test for missed attack stages before rollout Use realistic scenarios to check whether the model can connect beaconing, credential abuse, lateral movement, and data exfiltration into one coherent narrative.
• Keep human approval on response actions Separate narrative generation from containment steps so the model can assist triage without directly executing or authorising remediation.

Key takeaways

• The benchmark shows frontier models can recognise obvious SOC indicators, but that does not mean they can reconstruct an attack end to end.
• The most important failures were in later stages of the chain, where credential theft, lateral movement, and exfiltration determine whether an incident is understood correctly.
• SOC teams should evaluate agentic AI against complete investigation workflows and govern it like a controlled operating identity, not a standalone analyst.

Key terms

• Agentic SecOps: A security operations model in which AI systems can coordinate tasks and take bounded actions across alert handling, investigation, and response. The critical question is whether those actions are governed, explainable, and reversible enough to fit enterprise control requirements.
• Attack Chain Reconstruction: The act of linking events across time and controls to explain how an attacker moved from entry to impact. For modern fraud, reconstruction is essential because the meaningful threat often appears only after automated and manual stages are combined.
• Controlled Operating Identity: A non-human identity model for AI systems that perform operational work in security tooling. It defines what the system can access, what it may not do, and how its activity is logged so its output can be trusted and audited.
• Kill-Chain Coverage: The degree to which a detection or investigation workflow can trace all major stages of an attack, not just the initial alert. Good coverage helps teams avoid false confidence when a model or analyst sees only the first visible signal.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore the NHI Foundation Level course, the industry's only accredited NHI security programme, to connect identity governance with AI operating models and security workflows.