Security data fabrics are changing how teams balance SIEM and lakes

At a glance

What this is: This is an analysis of why many teams are moving from a SIEM-only model toward a security data fabric that combines SIEM and data lake capabilities.

Why it matters: It matters because SIEM cost, cloud-scale telemetry, and governance of retained security data now affect how IAM, NHI, and broader SOC teams decide what to ingest, enrich, and keep.

Context

A SIEM centralises security telemetry for real-time correlation, but it becomes expensive and rigid when log volume, cloud-native data, and retention demands grow faster than the platform can comfortably absorb. A security data lake solves scale and retention, but it shifts the burden to engineering and governance because raw data without structure is hard to query, govern, and operationalise.

The article’s core point is that the real decision is no longer SIEM versus lake. Teams are being pushed toward a pre-ingestion control plane where enrichment, filtering, and routing happen before data is committed to a high-cost or high-retention destination. That has a direct identity angle when pipelines carry authentication logs, service account activity, tokens, and other NHI telemetry that must remain both searchable and governed.

Key questions

Q: How should security teams evaluate security data lakes alongside SIEM investments?

A: Teams should evaluate them by control outcomes, not by storage cost alone. A security data lake is useful when it improves normalization, preserves investigative fidelity, and enables earlier detection logic without locking analytics into one vendor’s schema. If those outcomes are absent, the architecture may only shift costs rather than improve security operations.

Q: Why do security data fabrics reduce risk compared with SIEM-only designs?

A: They reduce risk by moving enrichment and routing decisions before ingestion, which preserves context while preventing every event from consuming premium SIEM resources. That makes it easier to keep high-value identity, cloud, and NHI telemetry searchable without forcing the whole environment into one expensive model.

Q: What breaks when telemetry is enriched only after ingestion?

A: When enrichment happens after ingestion, the SIEM already absorbs the full cost and the analyst gets context too late. That means routing decisions are made without enough information, sensitive data may already be stored, and triage slows because analysts must reconstruct context manually. Upstream enrichment avoids that waste.

Q: How should security teams stop a data lake from becoming a data swamp?

A: Start with governance, not volume. Define ownership, metadata standards, retention rules, and access boundaries before broad ingestion, then enforce schema validation and cataloging as part of the pipeline. A lake stays useful when analysts can find data, trust it, and query it without guessing which engine or dataset is authoritative.

Technical breakdown

Why SIEMs break at telemetry scale

A SIEM is optimised for indexed, high-value security events that need fast search, correlation, and alerting. Once every log source, cloud control-plane event, endpoint signal, and application event is forced through the same ingestion and retention model, the economics change and the system starts to rely on filtering, truncation, or short retention windows. That creates a trade-off between visibility and cost. The technical failure is not that SIEMs stop working, but that they become hardest to sustain precisely where telemetry volume is most useful for investigation and compliance.

Practical implication: Treat ingestion pressure as a control-design problem, not just a storage problem.

How security data lakes change detection architecture

A security data lake stores raw telemetry with schema-on-read, which preserves flexibility and enables long-term analytics, but it does not provide native real-time detection on its own. Detection must be layered on through query engines, custom analytics, or separate alerting systems. That makes the lake strong for hunting and forensics, but weaker as a standalone response surface. In identity-heavy environments, that matters because workload logs, authentication trails, and NHI activity often need both historical replay and near-real-time triage.

Practical implication: Use the lake for depth and retention, but do not mistake storage scale for operational detection capability.

What a security data fabric does differently

A security data fabric decouples collection, enrichment, routing, and analysis so each telemetry stream can be handled according to its value. This is where the article’s architecture becomes operationally important: enrichment and routing move upstream, so the platform can decide whether an event belongs in the SIEM, the lake, or both before ingestion locks in cost and retention. For identity telemetry, this creates a better place to govern service account events, delegated access records, and authentication anomalies because the routing decision can reflect context rather than raw volume alone.

Practical implication: Design the pipeline so context determines destination before ingestion charges and retention policies are fixed.

Threat narrative

Attacker objective: The practical objective is to hide malicious activity inside telemetry volume and exploit governance blind spots created by delayed enrichment and limited retention.

1. Entry occurs when high-volume telemetry reaches the security pipeline without enough context to distinguish routine events from high-value signals.
2. Credential and identity-related events become harder to separate from background noise when enrichment is deferred until after ingestion, which weakens prioritisation.
3. Operational impact follows when critical alerts, historical retention, or investigative depth are constrained by cost, queue pressure, or short retention windows.

Breaches seen in the wild

• Sumo Logic breach 2023: A compromised credential opened a Sumo Logic AWS account; customers were told to rotate API keys and stored credentials. No data impact was found.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Security data fabric is becoming a governance layer, not just an analytics pattern. The article shows that enrichment, filtering, and routing now decide what becomes searchable, retainable, and actionable before SIEM economics take over. That is a control-plane issue, not a storage preference. For identity teams, the same logic applies to authentication trails and NHI telemetry, where context must be attached before the event disappears into cost-driven truncation.

The old SIEM-versus-lake debate misses the real boundary, which is pre-ingestion decisioning. Teams are no longer choosing between speed and scale in the abstract. They are choosing where context is applied, where retention is committed, and which events deserve expensive real-time handling. That means governance must move upstream into the telemetry pipeline if the programme wants both detection quality and usable history.

Security data fabric creates a new control expectation for cloud and identity telemetry. If service account activity, API calls, and login events are routed without enrichment, practitioners are effectively asking downstream tools to infer intent from volume alone. That is too late for both cost control and investigation quality. The practitioner conclusion is simple: identity-rich telemetry should be classified before it is priced.

Cloud-scale security operations now depend on telemetry normalisation as much as on alert logic. Normalisation is no longer a reporting convenience because heterogeneous logs cannot support correlation, routing, or replay unless the pipeline standardises them early. For SOC and IAM teams, that means governance over field consistency, enrichment timing, and retention tiering is becoming part of the security architecture itself.

Security teams should expect architecture decisions to shape incident response outcomes. If the SIEM only receives a narrowed feed, the organisation may preserve cost efficiency but lose reconstruction fidelity when an investigation expands. The more distributed the environment becomes, the more the field needs upstream context management, because late enrichment cannot recover evidence that was never preserved in a usable form.

What this signals

Security data fabric is becoming the practical answer to telemetry overload. The programme issue is not whether a team owns a SIEM or a lake, but whether it can decide earlier what each event means and where it belongs. That makes upstream classification a governance control, not an engineering convenience.

Identity and NHI telemetry are especially exposed by late-routing architectures. Authentication logs, service account events, and delegated access records lose operational value quickly if context is attached after ingestion. For IAM and SOC teams, the question is now whether the pipeline can preserve meaning before storage economics strip it away.

For practitioners

• Define telemetry classes before ingestion Classify logs, alerts, and identity events by investigative value, retention need, and response urgency before they reach the SIEM or lake.
• Move enrichment into the pipeline Attach context such as asset identity, source classification, user or workload identity, and threat intelligence while events are still in motion.
• Route high-value events to the SIEM Keep recent, high-fidelity, and time-sensitive security events indexed for fast search while offloading bulk telemetry to lower-cost storage.
• Preserve identity telemetry for replay Ensure authentication, service account, and delegated access records remain searchable in the lake for hunting, forensics, and audit replay.
• Tie SOAR workflows to both layers Connect alerts and enriched findings from the SIEM and the lake to a single incident response workflow so escalation paths stay consistent.

Key takeaways

• Security data fabrics shift the important decision from storage selection to pre-ingestion governance of telemetry value.
• The architecture only works when enrichment, filtering, and routing happen before cost and retention are locked in.
• For identity-heavy environments, the practical test is whether authentication and NHI events remain both searchable and economically sustainable.

Key terms

• Security Data Fabric: A security data fabric is a neutral data layer that collects, transforms, routes, and sometimes filters telemetry before it reaches downstream tools. Its value is that it decouples data control from the SIEM, making migrations, parallel runs, and evidence preservation more manageable.
• Schema-on-read: Schema-on-read means data is stored first and structured later, at query time. For security teams, that allows many log formats to land in one place without upfront modelling, but it also means ingestion validation must catch format drift before detections and investigations start failing.
• Telemetry Enrichment: Telemetry enrichment is the process of adding context to raw security data so it is more useful for investigation and response. That can include asset, identity, geolocation, or threat intelligence context, but enrichment must be controlled so it does not distort the original evidence record.
• Tiered Retention: Tiered retention is the practice of storing security data in different locations or tiers based on how quickly it must be queried, how long it must be kept, and how much it costs to retain. It helps balance real-time detection needs with long-term forensic and compliance requirements.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management for practitioners who need to connect identity control to broader security architecture. It is designed for teams that must govern machine access as part of a wider security and risk programme.