ASD ACSC logging guidance shows the data layer is the SOC bottleneck

At a glance

What this is: A 15-agency logging advisory says effective detection depends on structured, trustworthy, and long-retained log data rather than simply more SIEM visibility.

Why it matters: SOC and IAM teams need to treat log quality as a control plane issue because detection, investigation, and identity abuse analysis all fail when timestamps, schemas, or source coverage are inconsistent.

By the numbers:

• It can take up to 18 months to discover a cyber security incident, according to Axoflow.
• Some malware dwells on the network 70 to 200 days before causing overt harm, according to Axoflow.

Context

Event logging is not just a storage problem. It is a governance problem because detection only works when telemetry is structured, time-synchronised, retained long enough, and available from the sources that matter most.

That matters for SOC operations and for identity-driven investigations, because authentication events, privilege changes, and command execution often become useful only if the underlying logs survive schema drift and retention gaps. The article argues that the bottleneck sits in the data layer, not in the alerting layer.

Key questions

Q: What breaks when event logs are not structured and time-synchronised?

A: Correlation breaks first. Analysts lose reliable joins between authentication, process, and network events, so the SOC sees isolated records instead of a sequence. That weakens alert fidelity, slows investigations, and makes identity abuse harder to prove because the session trail is incomplete or misordered.

Q: Why does log retention matter if the SOC already has a SIEM?

A: A SIEM cannot analyse telemetry that has already been deleted, tiered away incorrectly, or overwritten before discovery. Retention has to cover realistic dwell time, not just short operational or audit windows, because many incidents emerge long after the original intrusion.

Q: What are the signs that alert investigation is failing in a SOC?

A: Common signs include a large share of alerts going uninvestigated, analysts spending too much time on benign events, delayed response times, and recurring burnout. Another warning sign is inconsistent conclusions across similar alerts, which usually means analysis depends too heavily on individual skill. When those signals appear together, the SOC is likely overwhelmed rather than truly prioritizing risk.

Q: How should security teams harden centralized logging against tampering?

A: They should segment the log store, restrict delete and modify rights to a very small set of justified roles, and protect records in transit and at rest. The goal is to make evidence loss harder than maintaining the compromise, so response teams retain trustworthy telemetry.

Technical breakdown

Why structured event logging matters for detection quality

Structured logging means each event is emitted in a predictable schema, such as JSON or key-value format, with stable field names and ordering. That consistency lets SIEM and XDR platforms correlate identity, device, and process activity without brittle parsing. When logs vary by source or vendor, the detection pipeline loses context before analytics even start. The practical issue is not log volume alone, but whether the downstream system can reliably extract the fields that make correlation possible.

Practical implication: Standardise event schemas before tuning detections so analysts can query, correlate, and baseline across sources without parser debt.

Why UTC timestamps and field baselines are detection prerequisites

Time is a security control in logging. UTC timestamps, ISO 8601 formatting, synchronised time servers, and consistent fields such as event type, user ID, device ID, and command executed allow teams to reconstruct sequences across distributed systems. Without that baseline, detections become approximate narratives instead of evidence. For identity investigations, the difference between a precise session ID and a missing one can decide whether access abuse is visible or lost in ambiguity.

Practical implication: Enforce timestamp and field consistency across endpoints, identity systems, cloud control planes, and OT telemetry.

How centralized log integrity changes incident response

A centralized, segmented logging facility reduces the chance that attackers can erase their own traces after compromise. The guidance assumes adversaries will try to modify or delete local logs to blind response, so secure storage, access restriction, and cryptographic integrity become part of detection architecture. Long-term retention matters because many incidents are discovered late, well after the original compromise. The result is that retention policy, storage tiering, and tamper resistance become core SOC design choices, not back-office archiving details.

Practical implication: Move high-value logs into protected storage with restricted delete rights and retention matched to realistic dwell times.

Threat narrative

Attacker objective: The attacker objective is to operate inside the environment without triggering reliable detection while preserving freedom to move, persist, and erase evidence.

1. Entry often begins with legitimate access patterns or living-off-the-land activity that avoids obvious malware signals.
2. Credential access and privilege abuse become visible only if authentication, process, and command-line logs are preserved with enough fidelity to correlate sessions.
3. Impact is the loss of detection and response visibility, which lets attackers move laterally, clear traces, or remain undiscovered for months.

Breaches seen in the wild

• CISA Private-CISA GitHub leak 2026: A CISA contractor's public GitHub repo exposed AWS GovCloud admin keys, Artifactory credentials and plaintext passwords for six months.
• Sisense breach 2024: A credential in Sisense's GitLab reportedly opened S3 buckets of customer tokens, passwords and certificates; CISA urged a full reset.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

The real bottleneck is the data layer, not the SIEM. Detection platforms can only reason over what they receive, and log pipelines that distort schemas, timestamps, or field completeness degrade the entire SOC stack. That makes logging governance a prerequisite for detection quality, not a separate housekeeping task. Practitioners should treat telemetry fidelity as an operational control.

Structured logging is now an identity assurance issue as well as a security telemetry issue. When authentication events, session IDs, and command fields are missing or inconsistent, teams lose the ability to reconstruct who or what used an identity at a given time. That is especially damaging in environments with privileged accounts, service identities, and cloud control-plane activity. The broader lesson is that identity evidence is only as trustworthy as the log schema carrying it.

Log retention has to be designed against dwell time, not policy minimums. If incidents can remain hidden for 70 to 200 days, and some are discovered even later, retention that satisfies short audit windows still fails operationally. This is where the named concept of detection retention gap matters: the gap between how long telemetry is kept and how long attackers actually stay hidden. Practitioners should size retention to the threat, not the filing cabinet.

Centralized log protection is part of anti-evasion design. Attackers often target local log stores because deleting evidence is cheaper than defeating every detection rule. Segmenting the log facility, limiting delete rights, and protecting integrity at rest and in transit are therefore defensive controls, not storage preferences. Security teams should assume adversaries will try to blind investigations and build for tamper resistance accordingly.

What this signals

Detection retention gap: the operational problem is no longer whether logs exist, but whether they stay searchable long enough to cover real attacker dwell time. SOCs that optimise for short-term query speed while underinvesting in durable retention will keep missing the cases that matter most.

For practitioners, the immediate implication is that data engineering and identity telemetry governance belong in the same conversation. Authentication, privilege change, and command execution logs are the minimum evidence set for modern investigations, and they must survive schema drift, clock drift, and pipeline filtering.

For practitioners

• Standardize log schemas across priority sources Map critical systems to a single structured format, preserve stable field names, and remove source-specific drift before forwarding to analytics.
• Enforce UTC and synchronized time sources Use UTC everywhere, apply ISO 8601 timestamps, and validate one-directional time sync so investigations can reconstruct sequence accurately.
• Prioritize identity and control-plane logs first Collect authentication, permission change, admin configuration, and command execution telemetry before expanding to lower-value sources.
• Move retention decisions to the threat timeline Set hot, warm, and cold retention based on realistic dwell time, then verify the long-tail data remains searchable for late discovery cases.
• Segregate and harden centralized log storage Restrict delete and modify rights, segment the storage from general IT, and protect the pipeline with integrity controls in transit and at rest.

Key takeaways

• Structured logging is a detection requirement, not an administrative preference, because correlation fails when schemas and timestamps are inconsistent.
• The guidance reflects a threat reality in which incidents may remain undiscovered for months, so retention must match attacker dwell time rather than local policy convenience.
• SOC teams should harden the logging pipeline itself, because centralized integrity controls and identity-aware telemetry are what make investigations dependable.

Key terms

• Structured Logs: Structured logs are machine-readable records that keep the same field layout across components and actions. They make it easier to trace a security workflow from detection to response, correlate events with metrics or traces, and investigate failures. For automated response systems, structured logs are a core operational control.
• Detection Retention Gap: Detection retention gap is the period in which an attacker can remain active after the telemetry needed to expose them has already been discarded or moved out of reach. It reflects the mismatch between operational retention settings and real-world dwell time.
• Telemetry integrity: Telemetry integrity is the confidence that logs, metrics, and traces accurately reflect what happened. If an attacker can alter, redirect, or suppress telemetry, the security team may still see data, but it can no longer trust that data for investigation, detection, or compliance evidence.
• Tamper-Resistant Logs: Tamper-resistant logs are audit records protected so they cannot be altered without detection. They matter because compliance evidence loses value if an administrator, attacker, or integration can rewrite the history of file access after the fact.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, identity lifecycle, and secrets management. It helps security practitioners connect identity control design to the broader resilience and detection work their programmes depend on.