Why one LLM is not enough for an AI SOC

At a glance

What this is: This is an analysis of why a single LLM cannot reliably operate as an AI SOC, with the key finding that pre-processing, semantic normalization, and validation are necessary for useful outputs.

Why it matters: It matters to IAM and security teams because SOC data often includes identity logs, and inconsistent or unvalidated outputs can distort investigations, access decisions, and incident response.

👉 Read Exaforce's analysis of why one LLM is not enough for an AI SOC

Context

An AI SOC fails when teams treat the LLM as the system rather than as one component inside a wider workflow. The core problem is not whether the model can generate fluent answers, but whether the security data it consumes is normalized, trustworthy, and structured enough to support investigation across identity, cloud, and endpoint sources.

This is especially relevant where SOC telemetry intersects with IAM. Entra ID, Google Workspace, and Okta logs often describe similar identity events with different schemas, field names, and semantics. Without canonicalization, security teams risk feeding inconsistent identity evidence into automation, which weakens both analyst confidence and downstream response.

Key questions

Q: What breaks when an AI SOC relies on a single LLM?

A: A single LLM breaks down when it is expected to read raw security data, preserve context over long investigations, and produce reliable judgments without validation. SOC telemetry is too inconsistent and too sensitive for that assumption. Normalization, enrichment, and evidence checking must happen before and after the model, or the output becomes fluent but untrustworthy.

Q: Why do identity logs create problems for AI SOC automation?

A: Identity logs often represent the same action with different field names, schemas, and semantics across vendors. That makes direct model interpretation risky because the LLM may not know which fields are equivalent or which details are missing. Canonicalization is needed so access events can be compared and validated consistently.

Q: How can analysts tell whether AI-driven SOC automation is actually working?

A: Look beyond alert volume and measure whether the platform produces accurate incidents, preserves tenant context, and shortens time to closure without creating rework. If analysts still need to reconstruct the story manually, the automation is reducing noise but not truly improving operational control.

Q: Should SOC teams use LLMs for all detection and response tasks?

A: No. LLMs are better at summarising, explaining, and orchestrating than at statistical detection or truth verification. High-confidence anomaly detection, correlation, and numeric analysis should remain in specialised systems, with the model used to interpret those outputs and support analyst decision-making.

Technical breakdown

Why raw SOC logs break LLM reasoning

LLMs are pattern-matching systems, not truth engines. They can summarise text and simulate reasoning, but they do not inherently understand the structure or reliability of security telemetry. In SOC environments, that matters because logs arrive from different tools with different field names, timestamps, and event models. If those inputs are inconsistent, the model can misread the same identity event as different things, or lose key context across a long investigation. The result is not just poor accuracy, but unstable operational trust in the output.

Practical implication: normalise and canonicalise identity and security logs before they reach the model.

Why multi-agent SOC architectures still need validation

A multi-agent design improves coverage, but it also introduces handoff loss. Each time one agent passes findings to another, context can be compressed, simplified, or dropped. That is a problem in investigations where the sequence of small identity and access events matters more than any single alert. Evaluation agents and post-processing checks are therefore not optional add-ons. They are the controls that reduce hallucination risk, catch drift, and verify whether the model output is consistent with source evidence.

Practical implication: add validation agents and evidence checks between investigation stages, not just after the final response.

Why behavioural and time-series analysis must sit outside the LLM

SOC workflows depend heavily on numerical analysis, outlier detection, and behavioural baselining. LLMs are not designed to calculate trends or detect subtle deviations in user behaviour over time, which is why they should not be the layer that identifies statistical anomalies. Those signals belong in specialized analytics, such as UEBA or other detection models, which can then feed structured findings into the LLM for interpretation. That separation of duties is what makes the AI SOC operationally defensible.

Practical implication: keep anomaly detection in statistical systems and use the LLM for explanation, triage, and synthesis.

NHI Mgmt Group analysis

One LLM is not an AI SOC because security operations depend on structured truth, not fluent output. The article correctly identifies the central design error in many AI SOC discussions: assuming the model itself is the product. SOC value comes from the pipeline around the model, including normalization, enrichment, evaluation, and controlled handoff between tools. When identity logs and security events arrive in incompatible formats, the model cannot reliably infer meaning from syntax alone. Practitioners should treat the LLM as an interface layer, not the trust anchor.

Semantic normalization is the real control plane for AI SOC quality. The most important architectural move is not prompt engineering, but shaping raw telemetry into consistent schemas before inference. That is the point where logging becomes decisionable data, and where identity events from Entra ID, Okta, or Google Workspace become comparable across sources. Without that layer, the AI SOC inherits ambiguity at the exact point where it needs precision. Practitioners should invest in data semantics before expanding model use cases.

AI SOC governance debt: the longer teams delay semantic and validation layers, the more brittle their automation becomes. A model can appear useful in narrow demos while hiding structural weaknesses that only emerge under real incident volume. That creates governance debt, because later remediation requires reworking ingestion, schemas, evaluation, and exception handling at the same time. For identity-heavy environments, this debt is especially costly because access decisions depend on accurate event interpretation. Practitioners should design for auditability from the start.

Identity telemetry is where AI SOCs most visibly fail or succeed. The article's examples are IAM-centric for a reason: identity logs are semantically dense and operationally consequential. If the AI layer misreads a session start, sign-in success, or actor field, the downstream investigation may be wrong even when the model sounds confident. That makes identity data a useful stress test for AI SOC design. Practitioners should validate the pipeline first on identity sources before trusting broader automation.

What this signals

AI SOC programmes will be judged less by model sophistication and more by data discipline. The practical bottleneck is not whether an LLM can summarise an alert, but whether upstream telemetry is structured enough to survive inference and downstream audit. Teams that do not build a semantic layer will keep seeing impressive demos and unstable operations. For identity-heavy environments, that means investing in canonical event models and validation controls before scaling automation.

AI governance and SOC governance are converging. As LLMs move closer to investigation and response, security leaders need to treat model output as governed evidence, not just convenient commentary. That shifts attention toward lineage, verification, and exception handling across the whole pipeline. The organisations that operationalise this early will have a cleaner path to AI-assisted triage than those that bolt it on later.

Agentic workflows increase the need for control over inputs, not just outputs. When multiple agents share investigation tasks, the weakest point is often the handoff between stages rather than the final answer. That means the programme needs traceability for schemas, transformations, and evidence provenance. A trustworthy AI SOC is built by controlling the data path end to end, not by asking the model to compensate for upstream ambiguity.

For practitioners

• Normalize identity logs into a canonical schema Map Entra ID, Google Workspace, Okta, and other identity sources to a shared event model before LLM ingestion so the same action is represented consistently across platforms. This reduces semantic drift and makes correlation defensible across investigations.
• Insert evaluation gates between agents Use validation agents or deterministic checks to compare intermediate findings against source evidence before passing a case to the next stage. This helps catch hallucinations, context loss, and overconfident synthesis in multi-agent SOC workflows.
• Keep anomaly detection outside the LLM Run behavioural analytics, thresholding, and statistical detection in dedicated systems such as UEBA or other analytics engines, then feed the structured findings to the model for triage and explanation.
• Test the pipeline on identity telemetry first Use identity logs as the first proving ground for AI SOC design because their schema differences quickly expose normalization gaps, field mapping errors, and downstream reasoning weaknesses.

Key takeaways

• A single LLM cannot replace the control stack needed for an AI SOC because security data must be normalized, validated, and context-preserving before automation is reliable.
• Identity telemetry is a stress test for AI SOC design because inconsistent schemas and field names can turn a fluent answer into a wrong one.
• Practitioners should separate anomaly detection, semantic processing, and model reasoning so each layer does the job it is actually suited to perform.

Key terms

• Action Layer: The action layer is the point where an identity moves from asking for access to doing something with that access. For AI agents, this layer matters because tool use can happen faster than human review, and the meaningful risk appears when actions are chained across systems.
• Agent Orchestration: Agent orchestration is the coordination of multiple AI agents or workflows to complete a task set with limited human intervention. In identity terms, it creates delegated execution paths that need ownership, scope limits, and auditability because work is no longer performed only by a person in one session.
• Context Window: The context window is the text a model receives at one time, including prompts, retrieved documents, and conversation history. Security teams care about it because it becomes the practical boundary between trusted instructions and untrusted content, especially when the application assembles that text automatically.
• Hallucination: An AI-generated response that is fluent and plausible but incorrect, unsupported, or fabricated. For identity and governance teams, hallucination is a control issue because users may act on it as if it were trusted system output, especially when the chatbot sits inside an operational workflow.

What's in the full article

Exaforce's full blog covers the operational detail this post intentionally leaves for the source:

• Concrete examples of how the semantic layer normalizes Entra ID, Google Workspace, and Okta logs into a common structure.
• The multi-agent workflow design behind orchestrator, evaluation, and response agents in an AI SOC.
• Implementation details for pre-LLM data processing, including deduplication, sanitization, and schema definition.
• The specific reasoning limits the vendor associates with context windows, drift, and hallucination in SOC investigations.

👉 The full Exaforce post covers the semantic layer, multi-agent handoffs, and the limits of using LLMs alone in SOC workflows.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and workload identity for teams building identity-aware security controls. It helps practitioners connect identity discipline to broader security operations and governance.