Security data lake pipelines need governed normalization, not more ETL

At a glance

What this is: This is an analysis of security data lake pipeline management, with the key finding that telemetry becomes analytically useful only when normalization, routing, and delivery are governed end to end.

Why it matters: It matters because IAM, NHI, and broader security programmes depend on reliable telemetry pipelines to see credential abuse, privilege misuse, and control failures across cloud and application estates.

By the numbers:

• Only 44% of organisations have implemented any policies to manage their AI agents, despite 92% agreeing that governing AI agents is critical to enterprise security.
• Only 13% of organisations feel extremely prepared for the reality of agentic AI despite the majority racing toward autonomous adoption.

👉 Read TENZIR's analysis of security data lake pipeline management

Context

Security data lakes promise a single place to analyse telemetry, but that promise fails if the data arrives in incompatible formats, remains unnormalised, or cannot be delivered into the lake in a usable table structure. The primary governance problem is not storage capacity. It is whether the pipeline produces reliable, standardised security data fast enough for detection, investigation, and retention requirements.

For identity and access teams, the same issue applies to authentication, authorisation, and workload telemetry. If logs from service accounts, API keys, tokens, and privileged sessions are fragmented across formats, security teams lose the evidence needed to spot misuse and validate control effectiveness. The article sits in the cybersecurity-beyond-identity domain, but its operational relevance is strongest where security data quality underpins identity-led detection and response.

Key questions

Q: How should teams decide between a data lake and a data warehouse for security telemetry?

A: Use a lake when you need raw, multi-format data, historical replay, and flexible analytics across many sources. Use a warehouse when the data is already structured and the use case depends on strict schema enforcement at ingestion. Many security programmes need both, with clear rules for what lives where.

Q: What breaks when security logs are ingested without normalization?

A: Without normalization, fields do not line up across sources, detections become inconsistent, and analysts lose time reconciling incompatible records. The result is a lake that contains data but cannot reliably support cross-source correlation, which defeats the point of centralising telemetry in the first place.

Q: How do you know if a security data lake pipeline is actually working?

A: It is working when raw telemetry can be transformed into a stable schema, delivered into the target storage layout, catalogued correctly, and queried during real operational spikes. If schema changes, burst traffic, or destination-specific write rules regularly interrupt that flow, the pipeline is failing as a control.

Q: Who is accountable when a security pipeline drops critical telemetry?

A: The accountability sits with the team that approved the data reduction policy and with the platform owners who accepted the loss of security context. Governance frameworks should treat telemetry filtering as a control decision, because once evidence is dropped, investigation and compliance obligations can become much harder to satisfy.

Technical breakdown

Why security telemetry needs normalization before it reaches a lake

Raw security logs are rarely analysis-ready. Endpoints, cloud services, firewalls, and applications emit different structures, field names, and event semantics, so a security data pipeline has to translate them into a common schema such as OCSF before downstream analytics can work reliably. Normalization is not just format conversion. It is the process of making events comparable across sources so detections, retention policies, and investigation workflows do not break when one log source changes shape.

Practical implication: standardise the transformation layer before storage, or every downstream control will inherit schema drift.

How delivery mechanics shape the usability of a security data lake

Once data is normalised, the pipeline still has to deliver it into object storage, table formats, and catalog structures that the lake can query. That introduces concerns such as batching, partitioning, catalog refreshes, and destination-specific write patterns. These mechanics determine whether the lake is immediately searchable or quietly accumulating data that analysts cannot reliably query. In security operations, a technically complete ingest path is not enough if the destination layout prevents timely use.

Practical implication: treat catalog registration and table delivery as part of the control plane, not as an afterthought.

Why streaming execution matters for bursty security workloads

Security telemetry is uneven by nature. Threat feeds spike, cloud events arrive continuously, and incident periods can create sudden volume surges. A streaming-first pipeline with backpressure can process data incrementally while preventing memory overload or source collapse. That matters because batch-oriented ETL often introduces latency and fragility exactly when security teams need stable throughput. The architectural advantage is not speed alone. It is resilience under unpredictable load.

Practical implication: size pipeline architecture for burst handling and backpressure, or detection latency will rise during incidents.

NHI Mgmt Group analysis

Security data lake governance is really telemetry governance. The article is not about storage alone. It is about whether a security programme can transform fragmented, low-value logs into evidence that supports detection, investigation, and compliance. That aligns with NIST CSF 2.0 and NIST SP 800-53 because data quality, auditability, and monitoring are inseparable from control effectiveness. Practitioners should treat the pipeline as part of the security architecture, not as IT plumbing.

OCSF mapping reduces analytical friction, but it also creates a new governance obligation. A common schema helps teams correlate events across cloud, endpoint, and application sources, yet the mapping layer becomes a point of control failure if it is maintained ad hoc. This is where schema drift turns into evidence drift. Practitioners should put ownership, change control, and validation around the normalization layer so the lake remains trustworthy.

Pipeline performance is now a security control issue. When ingestion and delivery break under bursty workloads, detection and response lag behind attacker activity. That means the operational question is not whether data can eventually arrive, but whether it arrives in time to matter. Teams should evaluate their telemetry stack against real-world load patterns, not lab conditions.

Identity telemetry is the hidden dependency in security analytics. Service account activity, token use, privileged session events, and API authentication records are only useful if they are consistently normalised and routed. In practice, identity teams and SOC teams need the same pipeline discipline. Practitioners should ensure identity-relevant telemetry is mapped, retained, and queryable as a first-class security dataset.

Security data lakes are moving toward governed pipelines rather than bespoke ingestion projects. That is the right direction because the cost of maintaining brittle custom ETL rises as sources multiply and schemas change. The named concept here is telemetry pipeline sprawl, which is the point at which ingestion complexity becomes a security risk in its own right. Practitioners should reduce bespoke transformations and insist on standardised, reviewable pipeline governance.

What this signals

Security data pipeline quality is becoming a prerequisite for credible identity and threat operations. If service account, token, and privileged access events cannot be normalised and delivered reliably, identity teams will keep missing the evidence needed to spot misuse early.

Telemetry pipeline sprawl: when ingestion logic is scattered across bespoke ETL jobs, schema drift becomes a security problem, not just an engineering nuisance. That is why teams should align pipeline governance with NIST Cybersecurity Framework 2.0 and standardise the transformation layer before expanding the lake.

As AI agents, cloud services, and machine identities generate more security-relevant events, the value of the lake will depend less on raw volume and more on whether the pipeline preserves meaning at speed. Programme leaders should prepare for telemetry architecture to become a board-level dependency for detection and assurance.

For practitioners

• Standardise telemetry before storage Map raw logs to a common schema such as OCSF at the pipeline stage so detections and investigations use comparable event fields across sources.
• Treat delivery and catalog refresh as control steps Include partitioning, batching, and catalog registration in operational runbooks so data is not only written but also discoverable in the lake.
• Test streaming pipelines against burst conditions Validate backpressure, throughput, and memory stability using realistic surge patterns from cloud events and threat feeds, not only steady-state loads.
• Prioritise identity-relevant telemetry feeds Ensure service account, token, and privileged access events are consistently transformed and routed into the lake so identity misuse remains visible.

Key takeaways

• Security data lakes fail when telemetry is not normalized and governed before it reaches storage.
• The operational risk is not only missing data, but missing usable data at the moment analysts need it.
• Identity, SOC, and cloud teams should treat the telemetry pipeline as part of the security control plane.

Key terms

• Security Data Lake: A security data lake is a centralised repository for storing large volumes of security telemetry in a queryable form. Unlike a narrow SIEM pipeline, it is designed to keep heterogeneous logs accessible at scale so analysts and automation can correlate identity, endpoint, cloud, network, and application evidence.
• OCSF: The Open Cybersecurity Schema Framework is an open model for describing security events, findings, and objects in a common structure. It lets vendors map their telemetry into a shared language so downstream analytics, investigations, and automation can work across sources more consistently.
• Telemetry pipeline: A telemetry pipeline is the path security data follows from collection to analysis and retention. In mature environments it must preserve context, maintain throughput, and avoid introducing blind spots as sources, formats, and volumes change over time.
• Backpressure: Backpressure is the mechanism used to slow or control producers when consumers or brokers cannot safely absorb more traffic. In governance terms, it helps protect platform stability and prevents noisy or misbehaving clients from overwhelming shared event infrastructure.

What's in the full article

TENZIR's full article covers the operational detail this post intentionally leaves for the source:

• Step-by-step examples of how TQL normalises raw logs into OCSF events.
• Operator-level detail on routing normalised data into Amazon Security Lake, Snowflake, and parquet-based table layouts.
• The mechanics of batching, partitioning, and catalog refreshes that make lake delivery work at scale.
• How the pipeline engine handles streaming execution and backpressure under bursty security workloads.

👉 The full TENZIR article covers the normalization, routing, and streaming details behind the security data lake pipeline.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security and identity practitioners connect identity control to the broader programmes they operate.