Raw logs make AI triage fail before reasoning even starts

At a glance

What this is: Axoflow’s analysis shows that AI triage can fail consistently when agents are asked to judge raw, multi-format security logs without a shared schema or context.

Why it matters: For SOC and IAM-adjacent teams, this matters because access, identity, and endpoint signals only become actionable when data is normalized enough for reliable correlation and decisioning.

By the numbers:

• The agent spent 14 to 32 turns parsing and correlating 82 log lines before it could reach a verdict.
• A tighter system prompt reduced token spend by roughly 69% in the raw-log runs.

👉 Read Axoflow's analysis of why raw logs make AI triage fail

Context

AI triage fails when the system is asked to reason over raw telemetry before the data has been made interpretable. In practice, security logs arrive in incompatible formats, carry inconsistent field names, and often omit the context needed to distinguish a true incident from noise. The primary problem here is not model intelligence but data readiness, and that is also where identity and access signals become fragile if correlation depends on guesswork.

This is especially relevant for SOC teams, because identity events such as cloud key issuance, domain controller brute force attempts, and endpoint detections only become trustworthy when they can be aligned across sources. Without normalization, the AI agent becomes a parser first and an analyst second, which increases cost while reducing decision quality. That failure pattern is typical for raw-log workflows, not an edge case.

Key questions

Q: What breaks when AI triage is built on raw security logs?

A: The system loses decision quality before reasoning even starts. Raw logs force the agent to parse incompatible formats, infer field meaning, and reconstruct context on the fly, which increases cost and creates silent misclassification risk. Reliable AI triage needs normalised, enriched data first, then model-based analysis second.

Q: Why do identity and cloud context matter in AI SOC triage?

A: Because an IAM key, a service account, and a suspicious IP only mean something when the surrounding environment tells you whether they are expected. Without that context, the model guesses. Identity data, asset ownership, and reputation inputs turn raw events into defensible security signals.

Q: How do organisations know if AI triage is actually working?

A: Measure whether the AI improves high-fidelity detection, shortens time to verified response, and preserves reviewer trust in its decisions. A system that merely closes more alerts is not enough. The right signal is whether the SOC can validate its conclusions quickly and use them in real investigations without rework.

Q: Should organisations optimise token cost before fixing log quality?

A: No. Token cost matters only after the data is trustworthy enough to support a correct decision. If raw logs still need heavy parsing and context reconstruction, cheaper prompts simply make the wrong answer cheaper to produce. Fix normalization, enrichment, and correlation first.

Technical breakdown

Why raw logs force the agent into parsing before triage

Raw logs are not a shared decision format. A Cisco ASA event, a Windows Security 4625 record, a CrowdStrike-style JSON event, and an AWS CloudTrail entry each encode different fields, timestamps, and semantics, so the agent cannot compare them directly. It must first infer structure, map fields, deduplicate repeated events, and reconstruct a timeline. In that state, the AI is doing data engineering under uncertainty, not security reasoning. Practical implication: triage pipelines need normalization and enrichment before agentic analysis begins.

Practical implication: move schema mapping and correlation upstream of the agent so it sees structured events, not heterogeneous raw telemetry.

Why verdict quality collapses without context enrichment

A triage verdict depends on more than field extraction. The agent also needs environmental context such as whether an IP is a known scanner, what a service account normally does, and whether a cloud IAM key is expected in that workflow. Raw logs usually do not carry that context, so the model fills gaps with inference. That creates a hidden failure mode because the run completes successfully even when the answer is wrong. Practical implication: enrich identity, asset, and reputation context before any AI SOC decision layer.

Practical implication: attach asset intelligence, identity context, and reputation data before automated verdicting.

Why lower token cost can increase operational risk

Token optimisation is attractive because raw-log triage is expensive, but cheaper prompts do not fix missing structure. When the system is instructed to move faster and think less, it spends fewer tokens yet still lacks the data quality required for a correct call. That means cost reduction can directly amplify false confidence. In AI SOC design, efficiency only matters after the evidence is reliable. Practical implication: measure verdict accuracy and correlation quality before optimising prompt length or model spend.

Practical implication: treat token savings as secondary to correlation accuracy, false-positive rate, and explainability.

Threat narrative

Attacker objective: The objective is to create a noisy, multi-source incident pattern that conceals credential abuse and malware delivery long enough for defenders to misclassify it.

1. Entry begins with a known-scanner IP brute-forcing a domain controller while four benign IPs create distracting noise in the same window.
2. Credential abuse follows when the scenario includes minting a cloud IAM key for a service account, creating a privilege path beyond the initial login failure.
3. Impact occurs when the AI SOC cannot reliably distinguish malicious from benign activity and delivers a confident but wrong triage verdict.

NHI Mgmt Group analysis

AI triage becomes unreliable when raw telemetry is treated as decision-ready evidence. The core issue is not that the model cannot reason, but that the environment forces it to perform parsing, field mapping, and correlation before it can even decide what happened. That creates a fragile control plane where completion can be mistaken for correctness. For AI SOC programmes, the operating assumption that raw logs are suitable input is the first thing to challenge.

Identity data is only useful to an AI SOC when it is normalised into a trustworthy context layer. The scenario includes a cloud IAM key minted for a service account, which shows why identity signals cannot be isolated from asset, reputation, and behavioural context. This is where NHI governance intersects with SOC design: a credential event without normalisation is just a field, not a control signal. Teams should treat identity context as prerequisite infrastructure, not a post-processing add-on.

Cost optimisation without data governance creates a detection-response latency problem. The article shows that tightening prompts reduced token spend but increased wrong answers, which means the system became cheaper to operate while becoming less defensible. That is a governance debt pattern: the organisation pays less per run while increasing the probability of silent failure. The practical conclusion is that AI SOC design must optimise for verdict quality first, then for model efficiency.

Raw-log triage exposes a new control gap: parser dependency risk. When an AI agent must act as its own parser, every vendor format becomes a separate failure surface and every ambiguous field becomes an opportunity for guesswork. That is a different problem from model hallucination because the failure starts in the data layer. Practitioners should name this failure mode explicitly and remove it by design, not by prompt tuning.

Named concept: schema-before-reasoning. Security automation only becomes trustworthy when the data is normalised, enriched, and correlated before the AI decision layer sees it. The concept matters because it separates analytical intelligence from data plumbing and makes accountability measurable. For practitioners, schema-before-reasoning should become a design requirement for any AI SOC or agentic triage workflow.

What this signals

Schema-before-reasoning is the design rule this article makes hard to ignore: if the AI has to infer structure before it can triage, the control plane is already too late. For SOC leaders, that means the next maturity step is not a larger model but a more trustworthy data path, anchored by normalization, enrichment, and identity context.

The practical shift is toward deterministic preprocessing for logs and identity signals, with AI reserved for classification and prioritisation. That aligns better with security operations accountability because it creates measurable checkpoints before a verdict is issued, rather than after a confident guess has already been written to the console.

For practitioners

• Normalize logs before agentic triage Convert firewall, endpoint, and cloud telemetry into a shared schema before the AI sees it, then validate that identity and event fields survive the transformation intact.
• Enrich identity and asset context upstream Attach known-scanner reputation, service account context, and asset ownership data before verdicting so the agent is not guessing about whether activity is expected.
• Measure verdict quality, not completion rate Track correct triage rate, false positives, and correlation quality across raw and normalized inputs, because a completed run can still be a failed decision.
• Separate parsing from reasoning in the workflow design Use deterministic parsing and deduplication stages ahead of the model, then reserve the agent for classification, explanation, and prioritisation.

Key takeaways

• Raw telemetry can make AI triage fail even when the model is capable, because parsing and correlation become the hidden bottleneck.
• Identity and cloud context are not optional metadata in SOC automation, they are the difference between a signal and a guess.
• AI SOC programmes should optimise for verdict quality before token cost, or they risk accelerating wrong answers at scale.

Key terms

• Schema-before-reasoning: A design approach where security data is normalised, enriched, and correlated before an AI system is allowed to make a decision. It separates deterministic data preparation from probabilistic analysis so the model evaluates evidence rather than reconstructing it.
• Log Normalization: Log normalization is the process of converting different event formats into a common schema. It lets detection rules work across many sources, reduces source-specific rule drift, and keeps investigations consistent when infrastructure or vendors change.
• Detection-Response Latency: The elapsed time between identifying a security issue and executing a bounded, auditable fix. In data security programmes, long latency means exposure persists after discovery, which undermines the value of detection and weakens compliance evidence.

What's in the full article

Axoflow's full article covers the operational detail this post intentionally leaves for the source:

• The full methodology behind the nine-run verdict test, including how raw logs were formatted and correlated.
• The token-spend comparison between tighter prompts and fuller reasoning, including the cost delta in practice.
• The normalized-data variant of the same triage scenario, showing exactly where the answer changed.
• The step-by-step architecture difference between parser-heavy triage and structured event analysis.

👉 Axoflow's full post covers the test design, cost comparison, and the architecture change that improved triage quality.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, machine identity security, and identity lifecycle controls. It helps security practitioners connect identity discipline to the broader security programmes they operate.