Data lake architecture for security teams: avoiding the data swamp

At a glance

What this is: This is a guide to data lake architecture for security operations, and its key finding is that governance and storage design determine whether a lake stays queryable or becomes a data swamp.

Why it matters: It matters to IAM practitioners because the same access, catalog, and governance failures that break security data lakes also break visibility into identity and access activity across human, NHI, and workload identities.

By the numbers:

• AWS S3 has enforced default encryption since January 2023, with Azure and GCP offering equivalent defaults and optional customer-managed keys.

👉 Read Panther's guide to data lake architecture and security lakehouse design

Context

Data lake architecture is the design problem of keeping large volumes of raw security data searchable, governed, and affordable as the number of sources grows. In security operations, the challenge is not ingesting more data, but preventing schema drift, access bypass, and catalogue sprawl from undermining investigations. The same governance issues that affect telemetry also affect identity and access data, especially when human, NHI, and workload logs must remain consistent across tools.

Panther's article frames the problem around the security lakehouse pattern, but the underlying issue is broader: if governance is bolted on after ingestion, the lake loses evidential value and operational usefulness. That makes architecture decisions around object storage, metadata, and least-privilege access part of the security model, not just platform engineering. For identity-heavy programmes, this is typical of mature environments that need cross-domain observability rather than isolated log stores.

Key questions

Q: How should security teams stop a data lake from becoming a data swamp?

A: Start with governance, not volume. Define ownership, metadata standards, retention rules, and access boundaries before broad ingestion, then enforce schema validation and cataloging as part of the pipeline. A lake stays useful when analysts can find data, trust it, and query it without guessing which engine or dataset is authoritative.

Q: Why does access control need to sit at the storage layer in a security lake?

A: Because query-layer controls alone do not stop anyone with valid object-storage credentials from reading the data directly. Storage-layer controls let you enforce least privilege across all engines, which is essential when multiple tools, teams, and identities share the same lake.

Q: What breaks when schema drift is not managed in a security data lake?

A: Queries, detections, and dashboards can fail silently when a source changes field types or naming conventions. That turns a flexible lake into unreliable infrastructure, especially for investigations that depend on consistent timestamps, actor fields, and event context across many sources.

Q: How should teams decide between a data lake and a data warehouse for security telemetry?

A: Use a lake when you need raw, multi-format data, historical replay, and flexible analytics across many sources. Use a warehouse when the data is already structured and the use case depends on strict schema enforcement at ingestion. Many security programmes need both, with clear rules for what lives where.

Technical breakdown

Schema-on-read versus schema-on-write

A data lake stores data in its raw form and applies structure when the data is queried, which is why it can absorb many log formats without forcing upfront transformation. Schema-on-write, by contrast, validates records before storage. Security teams prefer schema-on-read for flexibility, but it creates a failure mode where source formats change and downstream queries silently break. That is why ingestion-time validation still matters even when the lake is designed for late binding.

Practical implication: validate critical fields at ingestion so schema drift does not break detections or investigations.

The five layers that keep a lake queryable

A functional data lake depends on ingestion, storage, processing, cataloging, and governance. Ingestion gets data in quickly, storage keeps it durable and low cost, processing normalises it, cataloging makes it searchable, and governance controls who can see what. If one layer is weak, the others have to compensate, and the result is usually slower investigations, more manual work, and higher operational risk.

Practical implication: review each layer separately rather than treating the lake as a single platform decision.

Why open table formats matter for security data

Open table formats such as Iceberg, Delta Lake, and Hudi add ACID behaviour, schema evolution, and time travel on top of object storage. For security teams, that means they can preserve raw evidence, replay historical data, and compare records across time without rebuilding the pipeline every time a source changes. The architectural benefit is not abstract portability, but the ability to keep forensic data reliable while multiple engines query the same storage.

Practical implication: choose open formats when investigations require historical reconstruction and multi-engine access.

NHI Mgmt Group analysis

Governance is the control plane, not an add-on. The article correctly shows that storage without cataloging and access control becomes a liability, not an asset. In security programmes, the same pattern appears when identity telemetry, detection data, and audit trails are separated from the controls that govern access to them. The practitioner takeaway is that governance must be designed alongside ingestion, not appended later.

Data lake swamp is a governance failure mode, not a storage problem. When security teams defer metadata standards, ownership, and retention rules, the lake becomes expensive to search and difficult to trust. That creates a visibility gap across both human and machine activity, including NHI-related logs that need consistent lineage. The practical conclusion is that queryability depends on policy discipline as much as on platform capacity.

Security observability now depends on identity-aware access to telemetry. A lake that multiple engines can query must enforce access at the storage layer, not just in the query engine, because valid object-store credentials can bypass higher-level controls. That matters for IAM, PAM, and NHI governance because analytics platforms often become shared trust zones. The practitioner implication is to treat log access as privileged access and govern it accordingly.

Medallion architecture gives security teams a usable evidence chain. Bronze, Silver, and Gold layers map cleanly to raw evidence, validated search, and detection-ready analytics. That progression reduces the risk of overwriting or over-normalising the records needed for investigations. The broader field implication is that architecture choices should preserve evidential integrity before they optimise reporting efficiency.

What this signals

Data lake architecture now shapes identity visibility, not just SOC performance. If security teams cannot catalog, classify, and govern their telemetry, they will also struggle to reconstruct human and non-human identity activity across investigations. That makes the lakehouse decision part of identity governance, because incomplete or inconsistent telemetry weakens every later control decision.

Queryability is becoming a governance requirement. The next phase of security operations will favour architectures that preserve raw evidence, enforce storage-layer access, and support historical replay without duplicating data across tools. For practitioners, the signal is clear: architecture that cannot support lineage and access discipline will not support resilient detection either.

The programme-level implication is that identity data, cloud telemetry, and detection data should be designed as one governed evidence layer. Teams that separate them too early will spend more time reconciling datasets than improving response quality, while teams that align them can use the same controls for access review, auditability, and incident reconstruction.

For practitioners

• Define ownership and metadata standards before scale-up Set naming, classification, and ownership rules before the first major ingestion wave, then make catalog registration mandatory for every new source. If analysts cannot find who owns a dataset, the lake is already drifting toward a data swamp.
• Enforce storage-layer access controls Apply IAM policies and table, column, and row permissions at the storage layer so query engines cannot become the only control boundary. Pair this with audit logging on object storage and sensitive datasets, especially where identity telemetry or NHI logs are involved.
• Use open table formats for forensic continuity Adopt Iceberg, Delta Lake, or Hudi where you need schema evolution, time travel, and repeated historical replay of security data. This preserves evidence across format changes and reduces dependence on one query engine for access to the same records.
• Treat schema drift as a detection risk Add ingestion validation for critical fields such as timestamp, actor, source system, and action so a changed field type does not silently break downstream queries. Route malformed records into quarantine tables instead of dropping them.

Key takeaways

• A security data lake fails when governance is treated as a later phase instead of a core design layer.
• Schema drift, storage-layer access gaps, and weak cataloging are the main reasons a lake turns into a swamp.
• The practical answer is to pair open table formats with storage-level least privilege and strict metadata discipline.

Key terms

• Schema-on-read: Schema-on-read means data is stored first and structured later, at query time. For security teams, that allows many log formats to land in one place without upfront modelling, but it also means ingestion validation must catch format drift before detections and investigations start failing.
• Medallion Architecture: A layered data design that separates raw ingestion, cleansed transformation, and curated analytics outputs. It gives teams clearer control points for quality, lineage, and reporting consistency, which is why it is widely used in governed data platforms.
• Open table format: An open table format is a storage layer that adds transaction handling, schema evolution, and historical replay on top of object storage. For security data, it helps teams keep evidence reliable across changing sources and lets different query engines use the same dataset without duplicate copies.
• Data swamp: A data swamp is a lake that has lost its searchability, trust, or governance because data arrived faster than the controls around it. In practice, it is not just messy storage. It is an operational failure where analysts cannot reliably find, verify, or use the data they are meant to protect.

What's in the full article

Panther's full blog post covers the operational detail this post intentionally leaves for the source:

• Implementation patterns for ingestion, storage, and processing layers in a security lakehouse
• Examples of Bronze, Silver, and Gold mappings for security detection workflows
• Design trade-offs between Spark, Flink, and Trino for different security use cases
• Governance and access-control details for object storage, cataloging, and audit logging

👉 Panther's full blog post covers the layer-by-layer design choices and governance practices behind a queryable security lake

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to the wider security programmes they operate and defend.