Data warehouse, lake, or lakehouse: the security trade-off

At a glance

What this is: Panther’s analysis says the security data architecture choice is really a trade-off between warehouse rigidity, lake flexibility and lakehouse governance.

Why it matters: IAM, SOC and security engineering teams need the right data layer because retention, investigation speed, schema change tolerance and access governance all shape detection and response outcomes.

Context

Security data architecture is a governance decision, not just an infrastructure choice. Where logs and telemetry land determines whether teams can search quickly, retain data economically and preserve trustworthy records as formats change.

Warehouses, lakes and lakehouses each answer that problem differently. For security teams, the real issue is whether a platform can absorb high-volume telemetry, support mixed workloads and keep controls strong enough for investigations, compliance and model training.

Key questions

Q: How should security teams choose between a data warehouse, lake, and lakehouse?

A: Choose based on workload mix, data shape, and governance demands. Warehouses fit structured analytics and reporting. Lakes fit raw, low-cost ingestion. Lakehouses are usually the better fit when the same data must support detection engineering, compliance, and machine learning without duplicating storage or losing consistency.

Q: Why do raw data lakes create problems for security investigations?

A: Raw lakes can accept almost any data, but without ACID transactions and metadata discipline they can return partial or untrusted results during active writes. That means an analyst may query incomplete evidence exactly when speed and fidelity matter most. The problem is not storage alone, it is the lack of governance around what is being read.

Q: What are the signs that a security data architecture is failing?

A: Common warning signs include log pipelines breaking when vendors change formats, repeated data copies across multiple platforms, rising retention bills, and slow searches during incident response. Those symptoms usually mean the architecture is optimised for ingestion or cost, but not for trustworthy investigation and audit use.

Q: When should teams move from separate data stores to a lakehouse model?

A: Consider the move when you are maintaining separate systems for SIEM ingestion, compliance reporting and ML training, because duplication usually means duplicated governance gaps too. A lakehouse becomes more compelling when you need one governed copy of telemetry that supports streaming, analytics and retention without constant replatforming.

Technical breakdown

Schema-on-write vs schema-on-read in security pipelines

A data warehouse requires fields to match a predefined schema before ingestion, which is why it performs well for structured analytics but breaks when telemetry formats change. A data lake does the opposite: it stores data first and applies structure later at query time. That flexibility helps with mixed log sources, but it also shifts the burden to governance, metadata and quality controls. For security data, the trade-off is direct. Rigid ingestion protects consistency but can reject useful evidence; flexible ingestion preserves evidence but can undermine trust if controls are weak.

Practical implication: Match schema strategy to the amount of log variability your detection and compliance workflows must absorb.

Why ACID transactions matter for telemetry and investigations

ACID transactions guarantee that readers see complete, consistent data even while writes are still in progress. Raw lakes lack that guarantee, so concurrent streaming and batch loads can leave analysts querying partially written datasets. That matters in security operations because threat hunting and breach reconstruction depend on a stable view of events, not a half-finished table. Open table formats such as Delta Lake, Iceberg and Hudi add transaction logs and metadata layers on top of object storage to close that gap without abandoning open storage.

Practical implication: Use ACID-capable table formats when analysts need trustworthy reads during active ingestion.

How lakehouse governance changes the security data model

A lakehouse keeps object-storage economics but adds warehouse-style governance through table formats, metadata and catalog controls. That lets teams run BI, ML and near real-time workloads on one copy of data instead of duplicating records across a SIEM, a lake and a warehouse. The architecture also improves forensic value because time travel can reconstruct what an analyst would have seen at a prior point in time. For security leaders, the key shift is from managing multiple disconnected repositories to governing one shared data layer.

Practical implication: Design governance around one authoritative security data copy instead of fragmenting controls across duplicate stores.

Threat narrative

Attacker objective: The practical objective is not a single exploit but operational blindness, where security teams cannot trust or quickly query the data they depend on.

1. Entry occurs when telemetry from changing sources lands in a rigid or unmanaged data platform that cannot reliably accept every field or record.
2. Escalation follows when analysts query partially written or inconsistently governed datasets and make decisions on incomplete evidence.
3. Impact is delayed investigations, weaker detection coverage and unreliable forensic reconstruction when teams need historical truth the most.

Breaches seen in the wild

• Snowflake breach: Snowflake breach compromised Ticketmaster, Santander and others via cloud credential abuse.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Data architecture is now a security governance control: security teams are no longer choosing only storage patterns, they are choosing how trustworthy their evidence layer will be under change. The article shows that schema handling, transaction integrity and retention economics all directly affect detection, investigation and auditability. Practitioners should treat platform design as an access-and-trust decision, not an IT plumbing decision.

Lakehouse governance is the most defensible middle ground for security telemetry: it preserves the low-cost storage model that modern log volumes demand while restoring enough structure for reliable analytics. That matters because security data is not static BI data, and a control plane that cannot reconcile streaming writes with consistent reads becomes a liability. The field should expect more consolidation around architectures that unify governance, query and ML on one copy of data.

Duplicate security datasets create governance drift: when the same telemetry is copied into separate SIEM, lake and warehouse estates, each copy develops different retention rules, access policies and freshness gaps. That makes consistent review and evidentiary reconstruction harder, not easier. The governance failure is fragmentation itself, and practitioners should standardise on a single authoritative data layer where possible.

Query latency is an operational control, not a convenience feature: the ability to search across terabytes quickly changes whether an investigation is still useful when the answer arrives. Slow, unpartitioned or poorly modelled storage delays containment and weakens threat hunting. Security teams should interpret performance as a response capability, not an engineering preference.

Open table formats are becoming the practical trust boundary: the important question is no longer whether the underlying storage is cheap, but whether the metadata layer can preserve consistency, history and governance at security scale. That is the architectural line practitioners should watch as lakehouse patterns mature.

What this signals

Unified security data layers are becoming a governance requirement: the more telemetry, compliance and ML all depend on the same evidence, the less defensible it is to manage those workflows in disconnected stores. Practitioners should plan for architectures that keep one authoritative copy of data under a shared control model.

Lakehouse design is shifting the security conversation from storage to trust: the decisive question is whether the platform can preserve consistency, provenance and query reliability while still supporting cloud-scale retention. That is where detection engineering, compliance and forensic readiness meet.

For practitioners

• Map security workloads to the right storage model Separate structured compliance reporting, mixed telemetry hunting and ML training workloads before choosing between warehouse, lake and lakehouse patterns.
• Prioritise ACID-enabled table formats for active telemetry Use Delta Lake, Iceberg or Hudi where concurrent ingestion and analyst queries must stay consistent during investigations.
• Consolidate duplicate security copies where possible Reduce separate SIEM, lake and warehouse copies of the same telemetry so retention, access policy and freshness do not drift apart.
• Treat data quality as detection infrastructure Validate schemas, field completeness and enrichment freshness before relying on AI-assisted triage or correlation rules.

Key takeaways

• Security data architecture affects more than cost, because it determines how reliably teams can detect, investigate and retain evidence.
• Warehouse rigidity, lake flexibility and lakehouse governance each solve a different part of the telemetry problem, but none is universal.
• The strongest design choice is the one that preserves trustworthy, queryable data under real-world ingestion and retention pressure.

Key terms

• Schema-on-Write: Schema-on-write requires data to match a predefined structure before it is stored. In security pipelines, that can protect consistency but also reject useful telemetry when upstream formats change unexpectedly.
• Schema-on-read: Schema-on-read means data is stored first and structured later, at query time. For security teams, that allows many log formats to land in one place without upfront modelling, but it also means ingestion validation must catch format drift before detections and investigations start failing.
• ACID Transactions: ACID transactions keep data changes atomic, consistent, isolated and durable. For security telemetry, they help prevent partially written or inconsistent datasets from undermining investigations during active ingestion.
• Open table format: An open table format is a storage layer that adds transaction handling, schema evolution, and historical replay on top of object storage. For security data, it helps teams keep evidence reliable across changing sources and lets different query engines use the same dataset without duplicate copies.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity and secrets management in practical terms. It helps practitioners connect identity control to the wider security programme they run every day.