Notifications
Clear all
Topic starter
15/08/2026 12:47 pm
TL;DR: Agentic SOCs can produce confident, polished triage narratives while still missing the evidence needed for a correct disposition, according to D3’s account of a BSides Pittsburgh talk by John Dwyer. The decisive control is not model confidence but whether investigations are evidence-led, replayable, and grounded in deterministic classification before the LLM summarizes anything.
NHIMG editorial — based on content published by D3: I Built an Agentic SOC in a Weekend. That's a Problem
Questions worth separating out
Q: How should security teams design agentic SOC workflows so the model does not guess too early?
A: Start with deterministic alert classification, then build an evidence graph, and only after that allow the model to generate hypotheses.
Q: Why do confidence scores fail as a measure of SOC accuracy in agentic investigations?
A: Confidence scores can describe how convincingly a system framed a conclusion, not whether the conclusion was supported by evidence.
Q: What breaks when an agentic SOC uses context before evidence?
A: The system can over-weight prior cases, documentation, or familiar patterns and misclassify a real intrusion as routine activity.
Practitioner guidance
- Define deterministic alert-genus routing Classify alerts into fixed investigative paths such as process execution or OAuth identity activity before any LLM summarisation begins.
- Require evidence graphs before disposition Force the platform to assemble process trees, auth events, and related artifacts into an evidence graph before it can label an incident.
- Test replayability with identical payloads Run the same alert through the system multiple times and compare the disposition, confidence, and evidence trace.
What's in the full article
D3's full article covers the operational detail this post intentionally leaves for the source:
- The side-by-side reasoning differences between LLM-led and evidence-led investigation flows
- The exact questions Dwyer recommends asking vendors about tool scope, replayability, and uncertainty handling
- The process classification and evidence graph approach used to keep context from overruling proof
- The practical SOC design considerations for reducing false confidence in agentic triage
👉 Read D3's analysis of evidence-led agentic SOC investigations →
Agentic SOC investigations: what changes when evidence comes first?
Explore further
15/08/2026 12:47 pm
Evidence-first is the only defensible architecture for agentic SOCs. An investigation platform that produces fluent narratives before it proves the underlying signal is optimising for persuasion, not security. That matters because SOC work depends on auditability, repeatability, and the ability to show why a disposition was reached. When the model is allowed to decide too early, the organisation gets an answer faster but a verdict it cannot defend.
A question worth separating out:
Q: Who is accountable when an agentic SOC misses a real intrusion?
A: The organisation is accountable for the workflow it allowed, not the model’s confidence level. Security, SOC, and platform owners need clear approval for tool scope, evidence thresholds, and escalation rules. Frameworks such as NIST CSF and NIST AI RMF support that shared governance model.
👉 Read our full editorial: Agentic SOCs need evidence-first rails, not confident summaries
