Notifications
Clear all
Topic starter
18/08/2026 2:38 pm
TL;DR: Security teams still spend too much effort normalizing messy logs into OCSF and delivering them into security data lakes, and TENZIR argues that pipeline tooling should absorb the ingestion, transformation, and routing burden rather than custom ETL. The real issue is governance at the data pipeline layer, where schema drift, backpressure, and destination-specific delivery decisions shape whether telemetry becomes usable security evidence or operational noise.
NHIMG editorial — based on content published by TENZIR: Why you shouldn’t build your security data lake yourself
By the numbers:
- Only 44% of organisations have implemented any policies to manage their AI agents, despite 92% agreeing that governing AI agents is critical to enterprise security.
- Only 13% of organisations feel extremely prepared for the reality of agentic AI despite the majority racing toward autonomous adoption.
Questions worth separating out
Q: How should teams decide between a data lake and a data warehouse for security telemetry?
A: Use a lake when you need raw, multi-format data, historical replay, and flexible analytics across many sources.
Q: What breaks when security logs are ingested without normalization?
A: Without normalization, fields do not line up across sources, detections become inconsistent, and analysts lose time reconciling incompatible records.
Q: How do you know if a security data lake pipeline is actually working?
A: It is working when raw telemetry can be transformed into a stable schema, delivered into the target storage layout, catalogued correctly, and queried during real operational spikes.
Practitioner guidance
- Standardise telemetry before storage Map raw logs to a common schema such as OCSF at the pipeline stage so detections and investigations use comparable event fields across sources.
- Treat delivery and catalog refresh as control steps Include partitioning, batching, and catalog registration in operational runbooks so data is not only written but also discoverable in the lake.
- Test streaming pipelines against burst conditions Validate backpressure, throughput, and memory stability using realistic surge patterns from cloud events and threat feeds, not only steady-state loads.
What's in the full article
TENZIR's full article covers the operational detail this post intentionally leaves for the source:
- Step-by-step examples of how TQL normalises raw logs into OCSF events.
- Operator-level detail on routing normalised data into Amazon Security Lake, Snowflake, and parquet-based table layouts.
- The mechanics of batching, partitioning, and catalog refreshes that make lake delivery work at scale.
- How the pipeline engine handles streaming execution and backpressure under bursty security workloads.
👉 Read TENZIR's analysis of security data lake pipeline management →
Security data lake pipelines: are your telemetry controls keeping up?
Explore further
18/08/2026 3:23 pm
Security data lake governance is really telemetry governance. The article is not about storage alone. It is about whether a security programme can transform fragmented, low-value logs into evidence that supports detection, investigation, and compliance. That aligns with NIST CSF 2.0 and NIST SP 800-53 because data quality, auditability, and monitoring are inseparable from control effectiveness. Practitioners should treat the pipeline as part of the security architecture, not as IT plumbing.
A question worth separating out:
Q: Who is accountable when a security pipeline drops critical telemetry?
A: The accountability sits with the team that approved the data reduction policy and with the platform owners who accepted the loss of security context. Governance frameworks should treat telemetry filtering as a control decision, because once evidence is dropped, investigation and compliance obligations can become much harder to satisfy.
👉 Read our full editorial: Security data lake pipelines need governed normalization, not more ETL
