Data warehouse, lake, or lakehouse: the security trade-off

At a glance

What this is: This is an analysis of how data warehouse, data lake, and data lakehouse architectures differ for security data, with the key finding that lakehouses are the strongest fit for mixed telemetry, governance, and real-time workloads.

Why it matters: It matters because identity, telemetry, and investigation data often include NHIs, cloud logs, and access events that security teams must retain, query, and govern without creating schema breakage or unmanageable cost.

By the numbers:

• Only 5.7% of organisations have full visibility into their service accounts.
• 92% of organisations expose NHIs to third parties, raising concerns about supply chain security.
• 71% of NHIs are not rotated within recommended time frames, increasing the risk of compromise over time.

👉 Read Panther's blog on data warehouse, lake, and lakehouse trade-offs for security teams

Context

Security data architecture is not just a storage choice. It determines whether teams can retain logs cheaply, query them quickly, and keep enough governance in place to trust the results when investigating incidents or feeding detection models. The primary keyword here is data lakehouse, because that is where the article’s core argument lands.

Warehouses, lakes, and lakehouses each solve a different part of the problem. Warehouses favour rigid structure and strong built-in controls, lakes favour low-cost ingest of varied data, and lakehouses try to combine both. For identity-heavy security programmes, the practical issue is whether telemetry tied to human and non-human identities stays queryable, governed, and usable under operational pressure.

This also intersects with identity governance because modern security telemetry increasingly includes service account activity, OAuth events, secrets-related audit logs, and workload identities. If those records are fragmented across multiple platforms, the programme inherits blind spots that affect both NHI control and human access investigation.

Key questions

Q: How should security teams choose between a data warehouse, lake, and lakehouse?

A: Choose based on workload mix, data shape, and governance demands. Warehouses fit structured analytics and reporting. Lakes fit raw, low-cost ingestion. Lakehouses are usually the better fit when the same data must support detection engineering, compliance, and machine learning without duplicating storage or losing consistency.

Q: Why does schema drift create governance risk in data pipelines?

A: Schema drift turns a functional change into a control problem because the pipeline may fail, silently drop fields, or alter downstream analytics without any explicit approval. The risk is not only data loss. It is also loss of traceability, because teams can no longer prove which schema version produced a given dataset.

Q: What breaks when security teams rely on raw data lakes alone?

A: Raw data lakes can become hard to trust when they lack transaction guarantees, metadata discipline, and consistent access controls. Analysts may see partial writes, inconsistent records, or poorly understood datasets. That undermines incident response, compliance reporting, and any AI model trained on the same data.

Q: How do you know if a lakehouse architecture is actually helping?

A: You should see fewer duplicate datasets, faster access to the same evidence across teams, and stable query results during active ingestion. If detection engineering, compliance, and ML consumers can use one governed data copy without creating retention conflicts, the architecture is doing its job.

Technical breakdown

Schema-on-write vs schema-on-read in security pipelines

A data warehouse enforces schema-on-write, which means records must match a predefined structure before ingestion. That gives fast SQL performance and strong consistency, but it also means log pipeline failures can occur when a source changes field names or event shapes. A data lake reverses that model with schema-on-read, allowing raw data to land first and structure to be applied later. That makes onboarding easier, but without metadata discipline and governance it becomes hard to trust what is stored or how complete it is. For security operations, the core question is whether your architecture tolerates source drift without silently losing coverage.

Practical implication: map your most volatile log sources to a storage model that can absorb schema drift without dropping fields.

Why ACID matters in threat hunting and investigations

ACID transactions provide atomicity, consistency, isolation, and durability. In a security context, that means a query should not read half-written telemetry while ingestion is still in progress. Raw data lakes do not provide native ACID guarantees, so concurrent writes from batch and streaming sources can produce partial results or inconsistent views during active investigations. Open table formats such as Delta Lake, Iceberg, and Hudi layer metadata and transaction logic on top of object storage so teams can query a stable dataset without sacrificing lake economics. This is especially important when analysts need reproducible evidence during incident response or audit reconstruction.

Practical implication: require transactionally consistent storage for datasets used in investigations, not just for reporting.

Lakehouse governance, time travel, and mixed workloads

A lakehouse is not just a cheaper warehouse or a better lake. It is a governance layer over object storage that supports BI, ML, and near real-time analytics from the same data copy. That matters because duplicating the same telemetry into a SIEM, a warehouse, and an ML store creates retention mismatch, inconsistent controls, and stale views of the same event. Time travel adds forensic value by letting teams inspect a table as it existed at a prior point in time, which is useful when analysts need to reconstruct what was visible before a detection or response decision. In other words, lakehouse value comes from unifying governance with flexibility.

Practical implication: prefer a single governed data copy when compliance, detection engineering, and machine learning all depend on the same events.

NHI Mgmt Group analysis

Data architecture is now a security control, not an infrastructure preference. The article’s central point is that where security data lives changes the quality of detection, forensics, and compliance evidence. A warehouse optimises curated analytics, a lake optimises raw ingestion, and a lakehouse is the first model that makes mixed security workloads tractable without forcing teams to choose between governance and flexibility. For identity-rich environments, that distinction matters because access logs, service account telemetry, and secrets events all need to stay queryable over time.

Schema rigidity creates hidden loss of security visibility. If a field rename or log-format change breaks ingestion, the control failure is not just technical inconvenience. It becomes a detection gap, especially when identity events move through cloud services, SaaS platforms, and CI/CD pipelines that change frequently. Schema drift exposure gap: this is the failure mode where a pipeline accepts the architecture but loses the evidence. Teams should treat coverage loss as a governance defect, not a formatting issue.

Lakehouse governance is the right answer when the same data must serve multiple teams. Security programmes rarely operate with one consumer. SOC analysts, threat hunters, compliance leads, and detection engineers often need the same record set under different latency and retention expectations. A unified table layer with ACID, time travel, and open formats reduces duplication while making auditability stronger. The practitioner conclusion is simple: if your programme relies on multiple copies of the same telemetry, the architecture is already costing you control.

Identity telemetry increases the need for durable storage semantics. Access events, OAuth logs, and service-account activity are only useful if they can be correlated across time and platforms. That is where lakehouse design becomes relevant to IAM and NHI governance, because the security programme cannot enforce lifecycle or investigate abuse against data it cannot reliably retain. The field should treat identity-related telemetry as evidence-grade data, not disposable observability noise.

Open formats are becoming the default compromise between platform control and operational portability. The article reflects a broader market shift toward object storage plus governance layers instead of proprietary, closed analytics stacks. That does not eliminate lock-in risk, but it reduces the penalty of changing engines or adding workloads later. For practitioners, the implication is to favour architecture choices that preserve data portability while still enforcing catalog, access, and retention controls.

What this signals

Schema drift exposure gap: security teams should assume that the most expensive failure in a data pipeline is not storage cost but lost evidence. When telemetry from SaaS, cloud, and identity systems changes shape faster than the pipeline can adapt, the programme loses visibility before it loses performance. That shifts architecture decisions into the domain of operational risk, not just platform preference.

For programmes that depend on service accounts, OAuth events, and access logs, the relevant design question is whether the evidence layer can survive change and still support investigation. NHI governance becomes harder when records are fragmented across tools, so teams should align telemetry architecture with lifecycle controls, retention policy, and auditability. The architecture either preserves the identity trail or erodes it.

The practical signal to watch is whether the same security event must be copied into multiple systems before anyone can use it. If so, the organisation is paying for duplication with weaker governance. Teams that standardise on a governed, open-format storage layer can usually improve traceability without sacrificing analytical flexibility.

For practitioners

• Classify telemetry by workload volatility Separate structured compliance data from high-churn security logs, then decide which sources need schema-on-read tolerance and which require schema enforcement at ingestion. This avoids forcing every feed into one warehouse-shaped pattern. Consider your most variable sources first, especially logs from cloud services and identity platforms.
• Require transactionally consistent storage for investigations Use open table formats or equivalent ACID-backed layers for datasets that analysts will query during active incidents. That ensures partial writes do not contaminate evidence views and makes historical reconstruction more defensible.
• Reduce duplicate security data copies Limit the spread of the same events across SIEM, warehouse, lake, and ML stores unless each copy has a clear control purpose. Duplicate datasets usually create different retention windows, inconsistent access controls, and more brittle audit trails.
• Treat identity telemetry as evidence-grade data Preserve access logs, service account activity, and OAuth events in a governed storage layer with reproducible history. That makes it easier to support investigations, validate lifecycle controls, and demonstrate accountability for NHI-related activity.

Key takeaways

• The core trade-off is no longer warehouse versus lake, but whether security data can stay governed without becoming brittle or expensive.
• Identity-heavy telemetry raises the stakes because service-account and access-event data only helps if it remains complete, queryable, and auditable over time.
• A lakehouse architecture is most defensible when one governed data copy has to serve detection, investigation, compliance, and machine learning at once.

Key terms

• Data Warehouse: A data warehouse is a structured analytical store built for fast querying and reporting. It enforces schema before data is loaded, which improves consistency and performance for stable datasets but can make changing security telemetry harder to ingest without pipeline changes.
• Data Lake: A data lake is a low-cost repository for raw data in object storage. It accepts structured, semi-structured, and unstructured data first, then applies structure at query time, which improves flexibility but requires stronger governance to prevent inconsistency and misuse.
• Data lakehouse: A data lakehouse is a cloud data architecture that combines the scale of object storage with the management features of a warehouse-like table layer. It supports AI and analytics workloads, but it also increases the importance of metadata integrity, recovery testing, and lifecycle governance.

What's in the full article

Panther's full blog covers the operational detail this post intentionally leaves for the source:

• A deeper comparison of schema-on-write and schema-on-read across security telemetry workflows
• Architecture examples for open table formats such as Delta Lake, Iceberg, and Hudi
• The medallion pattern in practice, including Bronze, Silver, and Gold layers
• Cost and retention trade-offs for SIEM, warehouse, and lakehouse deployments

👉 Panther's full post covers the architecture details, workload examples, and cost considerations behind the lakehouse model.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course, the industry's only accredited NHI security programme. It is designed for practitioners who need to connect identity governance to wider security operations and control programmes.