TL;DR: SIEM performance, retention and data ownership are determined first by how logs are stored, because teams must ingest continuously, search in minutes, preserve immutable evidence and absorb incident spikes while keeping costs under control, according to LogSeam. Open lake architectures reduce structural lock-in, but they shift pressure onto ingest, metadata and query planning.
At a glance
What this is: LogSeam explains that SIEM storage design drives whether logs stay fast to search, affordable to retain and usable as immutable evidence.
Why it matters: For IAM, NHI and security operations teams, the storage model affects whether identity, access and investigation data stays queryable quickly enough to support detection, forensics and long-retention governance.
Context
SIEM architecture is fundamentally a governance problem as much as a data engineering one. Security teams need logs that arrive continuously, remain searchable quickly, preserve evidence immutably and stay affordable as volume grows, while also supporting identity and access investigations across many systems.
LogSeam frames the main choice as a storage architecture decision rather than a detection problem. That matters to IAM and NHI practitioners because the same log data underpins identity investigations, privileged access review and workload activity analysis, and the storage layer determines whether that evidence is still usable when it is needed most.
Key questions
A: Detection latency, retention economics and evidence usability start to fail together. If logs cannot be queried quickly after landing, teams lose the window to act. If the storage model forces expensive scale-out, teams shorten retention or drop data, which weakens investigations and auditability later.
Q: Why do storage and compute separation matter for security log platforms?
A: They matter because coupled systems force every query and scale event to drag storage overhead with it. Separating them can improve portability and reduce stateful complexity, but only if the lake has strong ingest-time metadata, cataloging and file layout discipline to keep queries efficient.
Q: How do security teams know whether a SIEM lake is actually helping investigations?
A: Look for whether it can answer narrow identity or host questions quickly while still supporting broad historical hunts without exploding cost. If analysts must wait on scan-heavy queries, if retention keeps shrinking, or if cross-source correlation degrades, the architecture is not serving the investigation model.
Q: What should teams do when AI is used for SIEM investigations?
A: Treat AI as a consumer of governed context, not a replacement for it. Give the model access to identity resolution, access posture, and resource sensitivity so its conclusions are grounded in evidence. If it only sees raw logs, it will reproduce the same blind spots that already limit human analysts.
Technical breakdown
Why SIEM log storage is harder than general analytics
A SIEM has to satisfy contradictory requirements on the same dataset. It must ingest data without interruption, make recent events searchable in minutes, preserve years of immutable evidence and still support both narrow lookups and wide historical scans. That is harder than ordinary analytics because security data comes from many systems, in different vendor formats, with schema drift across releases. The storage layer therefore becomes part of the control plane for investigation speed, retention and evidence integrity, not just a backend repository.
Practical implication: design SIEM storage around incident response and audit needs, not just data volume.
What open lake storage changes for security data pipelines
An open lake separates storage from compute, which reduces the statefulness that makes document stores and engine-bound columnar systems expensive to scale. In the model described here, logs are written once as open Parquet in object storage and then queried by different engines as needed. The trade-off is that the lake must do more work at ingest and metadata time, because query speed now depends on file layout, catalog quality and statistics rather than on a tightly coupled index.
Practical implication: treat file layout, cataloging and partition discipline as security controls for searchability.
How ingest-time optimization affects detection latency
The article shows that the real performance gains come from pushing work left into ingestion. Time sorting, file statistics, bloom filters, dictionary encoding and semantic mapping all reduce how much data a query needs to touch. That matters because hunts, baselines and AI-driven analysis all compete for the same logs. If ingest does not prepare the data well, downstream searches become slower, costlier and less reliable, especially when analysts or agents need many small queries across long retention windows.
Practical implication: optimise ingest metadata so later investigations can skip data instead of scanning it.
NHI Mgmt Group analysis
SIEM storage is now an investigation-control decision, not a back-end choice. The article is right to treat storage architecture as the first SIEM decision because it shapes whether logs remain searchable, immutable and economically retained. For identity and security teams, that makes evidence quality dependent on data layout as much as on detections. Practitioners should evaluate log platforms on how they preserve investigative value over time, not just on ingestion throughput.
Open lake designs reduce lock-in, but they transfer responsibility to governance of the lake itself. Separating storage from compute can improve portability, yet it also means schema drift, file sizing, statistics and catalog quality become operational dependencies. That is especially relevant where identity and access telemetry must remain usable across tools and teams. Practitioners should assume the lake needs active governance if it is to support audit and response at scale.
Identity telemetry becomes more valuable when the storage layer preserves context across sources. The article's emphasis on heterogeneous log formats maps directly to IAM and NHI programmes that rely on cross-system correlation. If identity providers, cloud services and endpoints are not normalised well enough to search together, investigations fragment. Practitioners should prioritise architectures that preserve joinability across identity events, not just raw retention.
AI changes SIEM consumption patterns, which changes storage economics. When an agent can run hundreds of small searches continuously, human-sized search assumptions stop holding. That creates a new governance gap: platforms sized for occasional analyst queries may underperform when machine consumers become persistent. Practitioners should reassess search concurrency, query cost and response compactness before AI-driven workflows are layered onto the SIEM.
What this signals
Log storage policy is now a security governance issue. When the same dataset must support detection, forensics, compliance and machine-driven analysis, the architecture has to preserve both speed and evidentiary integrity. Teams that leave this decision to platform convenience usually inherit the wrong trade-offs later.
Searchability is the real control boundary. If identity and security logs cannot be joined, filtered and retained in usable form, the programme has data but not evidence. That is why SIEM design should be reviewed alongside IAM, NHI and incident response requirements, not after them.
For practitioners
- Define SIEM retention goals before choosing storage architecture Set evidence retention, search latency and cost targets up front, then test whether the storage model can satisfy all three under incident load and long-tail investigations.
- Validate searchability across identity and cloud log sources Confirm that logs from identity providers, endpoints, firewalls and cloud services can be queried together even when their native formats and schemas differ.
- Measure ingest-time optimization as a security control Check whether time sorting, file statistics, bloom filters and dictionary compression are actually reducing scan volume and query latency in production.
- Plan for AI-scale query concurrency Load-test the platform with many small parallel queries, not just a few analyst searches, so machine-driven investigations do not overwhelm the same lake.
- Keep logs in open, portable formats Store evidence in formats other tools can read so investigations are not trapped inside one engine or one vendor-specific storage shape.
Key takeaways
- SIEM storage design determines whether logs remain searchable, affordable and evidentiary over time.
- Open lake architectures reduce structural lock-in, but they make ingest-time metadata and file layout more important.
- Identity and security teams should treat log searchability as a governance requirement, not a backend preference.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK addresses the attack and risk surface, while NIST CSF 2.0, CIS Controls v8 and CSA Cloud Controls Matrix set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.DS-01 — Data-at-Rest Protection | Logs must stay protected, immutable and retained for investigations. |
| DE.CM-09 — Network Monitoring | The article is about continuously collecting and searching security telemetry at scale. | |
| Recommendation — Protect retained log data with controls that preserve integrity and availability across the full retention window. Tune monitoring architecture so telemetry remains queryable fast enough for detection and response. | ||
| CIS Controls v8 | CIS-8 — Audit Log Management | The core topic is how to collect, retain and analyse logs effectively. |
| Recommendation — Centralise audit log handling and validate retention, search and integrity across key log sources. | ||
| MITRE ATT&CK | TA0007;TA0009 — Discovery; Collection | SIEM data underpins threat hunting and investigative collection after suspicious activity. |
| Recommendation — Map hunt workflows to discovery and collection tactics so log coverage supports investigations. | ||
| CSA Cloud Controls Matrix | SEF — Security Event and Logging | The article directly concerns cloud-based security event storage and analysis. |
| Recommendation — Apply SEF controls to ensure cloud log retention, searchability and evidence integrity remain reliable. | ||
Key terms
- Security Data Lake: A security data lake is a centralised repository for storing large volumes of security telemetry in a queryable form. Unlike a narrow SIEM pipeline, it is designed to keep heterogeneous logs accessible at scale so analysts and automation can correlate identity, endpoint, cloud, network, and application evidence.
- Schema Drift: Schema drift is the mismatch between the attributes an IdP sends and the fields an application can store or interpret. It often appears as missing custom fields, inconsistent group data, or varying attribute names, and it undermines the reliability of lifecycle automation even when the core protocol works.
- Immutable Log: An immutable log is a record of events designed to resist alteration after creation. In identity governance, it supports non-repudiation by preserving who did what, when, and under what policy or context, so the organisation can reconstruct events during audit, fraud review, or dispute handling.
- Ingest-Time Optimization: Ingest-time optimization is the practice of preparing data as it arrives so later queries touch less of it. In SIEM and log lake architectures, that means sorting, compressing, indexing and enriching files early enough to reduce search cost and response latency later.
Deepen your knowledge
The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security and secrets management. It helps practitioners connect identity controls to broader security operations and resilience planning.
Published by the NHIMG editorial team on October 14, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org