Security data lakes can reduce SIEM storage without cutover risk

At a glance

What this is: Anomali argues that a security data lake can sit beneath an existing SIEM, taking high-volume telemetry out of expensive ingest storage while preserving detections and searchability.

Why it matters: For SOC and security architecture teams, this reframes SIEM modernization as a routing and enrichment problem rather than a full replacement project, with direct implications for cost, detection fidelity, and operational continuity.

By the numbers:

• A 100 GB/day Splunk deployment lists around $76,000 a year for the base platform before Enterprise Security is added, and the security add-on roughly doubles that.
• OCSF lists more than 900 contributors and roughly 200 participating organizations across its community.

👉 Read Anomali's analysis of security data lakes and SIEM storage

Context

Security teams often inherit SIEM environments that have become storage engines as much as detection platforms. When every source is ingested at full cost, the result is budget pressure, parser maintenance, and migration risk that make wholesale replacement harder than most teams can absorb. This article is primarily about reducing that burden without losing the detections already in place.

The identity angle is indirect but real: telemetry routing, enrichment, and correlation determine how well organisations can trace privileged activity, service-account misuse, and other access patterns across cloud and infrastructure. For teams managing IAM, PAM, and NHI signals inside the SOC, the governance question is not only what to alert on, but where the data should live so that detection remains economical and queryable.

Key questions

Q: How should security teams reduce SIEM ingestion costs without losing detection value?

A: Teams should move collection, classification, and normalisation upstream so the SIEM receives only the data needed for detection and investigation. That means defining retention tiers, dropping low-value noise, and preserving full-fidelity events for cases that support forensics, compliance, or identity investigations. Cost control works when it is a governance decision, not a storage cleanup exercise.

Q: Why do security data lakes help SOC teams more than simply buying a bigger SIEM?

A: Because the core problem is usually storage economics, not detection ambition. A data lake can absorb DNS, firewall, flow, and endpoint volume, then normalize and enrich the records so the SIEM only carries the content that drives investigations. That reduces cost without sacrificing correlation depth.

Q: What do teams get wrong when they try to modernize SIEM architecture?

A: They often treat migration as a cutover project instead of a phased routing change. That creates parser rewrites, rule migration, and parallel-run risk all at once. A better pattern is to move one source and one detection at a time, proving parity before the old ingest path is retired.

Q: How do OCSF and a data lake change investigation workflows?

A: They give teams a common event model across sources, which makes cross-platform correlation easier and reduces the need to rewrite detections for each log type. Investigators can query lake data alongside hot SIEM records, so the architecture supports both cost control and retained context.

Technical breakdown

Why SIEM storage becomes the bottleneck

SIEM platforms are optimised for correlation and alerting, not as low-cost repositories for all telemetry. When high-volume sources such as DNS, firewall, flow, and endpoint logs are forced into ingest-priced storage, the platform becomes expensive before it becomes more effective. Teams then face parser rewrites, detection migration, and parallel-run complexity if they try to replace the SIEM outright. The operational problem is architectural mismatch: the storage layer is doing work the detection layer does not need.

Practical implication: separate high-volume telemetry from high-value detections before ingest costs force a rewrite.

How a normalized data lake changes correlation

A normalized lake can absorb raw telemetry, standardize it to OCSF, deduplicate records, and enrich data at ingest so it remains searchable alongside the SIEM. OCSF matters because a field means the same thing across firewall, proxy, and cloud sources, which reduces rewrite work and makes correlation portable. This is not a visibility trade-off if implemented correctly. It is a routing decision that preserves analytical context while reducing storage pressure in the SIEM.

Practical implication: normalize once and route by signal value so correlation logic can survive source changes.

Why parallel migration is safer than cutover

The strangler-fig pattern avoids a hard cutover by moving one source and one detection at a time. That matters because detection parity can be tested incrementally, beginning with high-volume sources whose value is mostly in storage, not live alerting. DNS is a common first candidate because it often consumes substantial ingest while supporting a limited number of detections. The benefit is controlled de-risking: teams validate behaviour before they retire the old ingest path.

Practical implication: migrate source by source and retire each SIEM feed only after parity is proven.

NHI Mgmt Group analysis

Security data lake architecture is becoming a governance decision, not just a tooling decision. Once storage cost starts dictating what telemetry can be retained, detection strategy is constrained by economics rather than risk. That changes how SOC, platform, and identity teams should think about evidence retention, especially where privileged and non-human identity activity must remain searchable across multiple sources. Practitioners should treat data placement as part of control design, not a downstream infrastructure choice.

High-volume telemetry creates a false sense of completeness when it is stored inefficiently. Teams often assume they need a bigger SIEM when the real issue is that the wrong data is sitting in the wrong layer. Normalization, deduplication, and enrichment in a lake preserve investigative depth without forcing the SIEM to hold low-signal noise. The broader discipline shift is toward selective ingestion and governed retention, not universal routing into one platform.

OCSF-style normalization is the enabling concept here: portable detection content depends on a shared event model. Without a common schema, routing telemetry away from the SIEM risks fragmenting investigations and creating source-specific query logic. With it, teams can decouple collection from correlation and preserve consistency across firewall, DNS, cloud, and endpoint data. The practitioner conclusion is straightforward: standardize the data model before you change the storage architecture.

For identity-heavy environments, the real payoff is better evidence around access behaviour and privilege misuse. When service account, workload, and administrative activity can be retained and queried economically, SOC teams can correlate identity signals with network and endpoint telemetry more effectively. That supports stronger accountability for IAM and PAM programmes, because the evidence layer no longer collapses under ingest cost pressure. Practitioners should align logging architecture with identity investigation needs, not just volume management.

Agentic SOC operations will depend on the enriched dataset this model creates. The article correctly points out that the same architecture used for cost control becomes the foundation for later machine-assisted reasoning. That means teams designing for future AI use cases should focus on schema quality, enrichment policy, and retention boundaries now. Practitioners should view the lake as a governed data layer for both human and automated analysis.

What this signals

Data placement is becoming a control-plane decision for SOC programmes. As telemetry volumes keep rising, the teams that win operationally will be those that can separate storage economics from detection quality. That means budgeting for normalization, retention policy, and query performance together, rather than treating SIEM cost as a separate procurement issue.

Identity telemetry deserves the same routing discipline as infrastructure telemetry. Privileged access activity, service-account behaviour, and workload identity signals lose value if they are trapped in expensive ingest tiers or scattered across incompatible schemas. Teams should align SIEM and data-lake design with IAM and PAM investigation needs, not just generic log management.

High-fidelity data pipelines are now the prerequisite for agentic analysis, because machine reasoning is only as good as the evidence layer beneath it. The architectural lesson is to build the normalized, deduplicated, enriched dataset first, then layer automation on top. NHI Lifecycle Management Guide remains relevant wherever identity evidence must be retained across provisioning, rotation, and offboarding events.

For practitioners

• Map telemetry by signal value Classify sources such as DNS, firewall, flow, endpoint, and cloud logs by how often they contribute to detections versus how much ingest they consume. Move low-signal, high-volume data to a lake first so the SIEM retains the content analysts actually use. Suggested sequencing is especially important where privileged access events must remain searchable across both layers.
• Normalize events to a shared schema Adopt OCSF or an equivalent common model before shifting storage boundaries, so a blocked connection or identity-related event is represented consistently across sources. This prevents source-specific query sprawl and keeps correlation logic portable when you change where the data lands.
• Migrate one source at a time Use a strangler-fig rollout and validate detection parity before removing each SIEM ingest path. Start with a source that is expensive to store but relatively low value for live alerting, then confirm the SIEM still fires the same detections from lake data.
• Build the business case from your own numbers Quantify ingest volume reduced, retention cost avoided, and analyst hours recovered from reduced noise. Keep the calculation grounded in your own environment so the decision reflects actual routing and storage patterns rather than a generic vendor benchmark.

Key takeaways

• The article frames SIEM modernisation as a routing problem, with a data lake absorbing high-volume telemetry while preserving existing detections.
• The strongest evidence is economic and operational, not theoretical: ingest-based pricing, schema normalization, and phased migration all shape the architecture choice.
• Teams should move source by source, validate parity, and treat data model standardization as the prerequisite for both cost control and future automation.

Key terms

• Security Data Lake: A security data lake is a centralised repository for storing large volumes of security telemetry in a queryable form. Unlike a narrow SIEM pipeline, it is designed to keep heterogeneous logs accessible at scale so analysts and automation can correlate identity, endpoint, cloud, network, and application evidence.
• OCSF: The Open Cybersecurity Schema Framework is an open model for describing security events, findings, and objects in a common structure. It lets vendors map their telemetry into a shared language so downstream analytics, investigations, and automation can work across sources more consistently.
• Strangler-fig Migration: Strangler-fig migration is a phased replacement pattern where a new architecture is built beside the old one and workloads move gradually until the legacy path can be retired. For security data platforms, it lowers cutover risk by validating each source and detection before decommissioning the old feed.
• Telemetry Enrichment: Telemetry enrichment is the process of adding context to raw security data so it is more useful for investigation and response. That can include asset, identity, geolocation, or threat intelligence context, but enrichment must be controlled so it does not distort the original evidence record.

What's in the full article

Anomali's full article covers the operational detail this post intentionally leaves for the source:

• Side-by-side storage cost logic for SIEM ingest versus lake retention
• Stepwise strangler-fig migration sequence for moving logs without a cutover weekend
• Practical examples of OCSF normalization across firewall, DNS, proxy, and cloud events
• How to size the business case using ingest reduction, retention savings, and analyst time recovered

👉 Anomali's full post covers the migration sequence, OCSF handling, and cost case in more operational detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners align identity controls with the operational demands of modern security programmes.