Notifications
Clear all
Topic starter
02/08/2026 11:36 am
TL;DR: Legacy SIEM architectures built around sequential workers and opaque pipelines struggle as log volumes rise at 30% or more year on year, according to DataBahn’s analysis of enterprise telemetry processing. The practical issue is not just storage cost but the inability to scale collection, transformation, and correlation independently as new sources come into scope.
NHIMG editorial — based on content published by DataBahn: why legacy SIEMs are a problem and how cloud-native log pipelines scale
By the numbers:
- Log volumes are continually increasing, with typical growth rates of at least 30% year on year for existing sources.
- DataBahn has a number of customers in production processing 10s to 100s of Terabytes per day.
Questions worth separating out
Q: How should security teams scale telemetry pipelines without losing correlation quality?
A: Teams should decouple pipeline stages, use stateless services where possible, and preserve context across a durable transport layer.
Q: Why do worker-based SIEM pipelines fail as log volumes grow?
A: Worker-based designs fail because the same processing path handles multiple stages sequentially, so any hotspot slows the whole system.
Q: What do security teams get wrong about alert enrichment?
A: They often treat enrichment as a reporting layer rather than an investigation control.
Practitioner guidance
- Map telemetry bottlenecks to specific security outcomes Identify where sequential processing, load balancing, or opaque failure states are reducing visibility for identity, cloud, and endpoint events.
- Decouple collection, transformation, and delivery stages Move toward independently scalable services with a durable message layer so spikes in one source type do not stall unrelated streams.
- Shift enrichment ahead of SIEM ingestion Attach identity, asset, and threat context before events reach the SIEM so routing can separate high-value signals from routine telemetry.
What's in the full article
DataBahn's full article covers the operational detail this post intentionally leaves for the source:
- The specific microservice and Kafka pipeline design used to separate collection, transformation, and delivery stages.
- The production-scale health monitoring approach that detects schema drift, silent devices, and processing failures.
- The examples of autoscaling and self-healing behaviour under short-term spikes and long-term growth.
- The detailed explanation of how pre-ingestion enrichment changes SIEM cost and routing decisions.
👉 Read DataBahn's analysis of legacy SIEM scale and cloud-native log processing →
SIEM scale and pipeline bottlenecks: what practitioners need to know?
Explore further
02/08/2026 12:49 pm
Legacy SIEM scale is now an identity governance problem, not only a data engineering problem. Telemetry that cannot be ingested, correlated, and retained reliably weakens the control plane that detects account misuse, privilege escalation, and anomalous service-account activity. Identity and access programmes depend on evidence density, and brittle pipelines reduce that evidence before analysts ever see it. The practical conclusion is that SIEM architecture directly affects identity risk visibility.
A question worth separating out:
Q: How do you know if telemetry pipeline health is affecting incident response?
A: Look for unexplained lag, source silence, repeated schema failures, and situations where analysts cannot tell whether data was delayed or lost. If these symptoms persist, the pipeline is no longer just an infrastructure layer. It is actively shaping what the SOC can detect and when it can act.
👉 Read our full editorial: Legacy SIEM scale is breaking under growing telemetry volumes
