Should organisations replace SIEMs with security data pipelines?

Why This Matters for Security Teams

The question is less about replacing one tool with another and more about whether the organisation has a defensible way to manage telemetry, detection, and investigation at scale. A SIEM is still valuable for correlation, retention, and casework, but it becomes expensive and noisy when it receives everything indiscriminately. Security data pipelines help teams filter, enrich, normalise, and route events before they reach the SIEM, which improves signal quality and reduces avoidable storage costs.

This matters because modern environments produce high-volume data from cloud logs, endpoint agents, SaaS platforms, identity systems, and increasingly autonomous software. If telemetry is not curated upstream, analysts spend more time triaging low-value events than investigating credible threats. Control guidance in NIST SP 800-53 Rev 5 Security and Privacy Controls reinforces the need for auditability, monitoring, and event management, but it does not imply that every raw event belongs in the SIEM.

In practice, many security teams discover the limitations of their SIEM only after storage costs have risen, detections have become brittle, and investigation workflows have already been slowed by noise.

How It Works in Practice

A security data pipeline typically sits between sources and downstream consumers. Its job is to ingest events, transform them, and then decide which records should flow to the SIEM, which should go to lower-cost storage, and which should be suppressed altogether. That decision can be based on source type, severity, business context, identity risk, or detection logic. The goal is not to hide data, but to make sure the right data is available in the right place at the right time.

In mature designs, the pipeline performs several practical functions:

• Normalises schema so disparate sources can be searched and correlated consistently.
• Enriches records with asset, identity, or threat context before indexing.
• Deduplicates repetitive telemetry that adds cost without improving detection.
• Routes high-fidelity events to the SIEM while sending bulk logs to object storage or a data lake.
• Preserves chain of custody and timestamp integrity so investigators can trust the record.

This approach aligns well with the operational direction in CIS Critical Security Controls, especially where asset visibility, logging, and monitoring depend on data quality rather than raw volume. It also supports investigation workflows that need correlation across endpoint, cloud, and identity sources without forcing every event into a monolithic retention model. For organisations with autonomous agents or AI-assisted operations, the pipeline should also preserve tool-use events, decision traces, and privileged actions so non-human behaviour can be investigated like any other actor.

Security teams should treat the SIEM as one consumer in a broader telemetry architecture, not the only destination for all security data. These controls tend to break down when log sources are inconsistent, enrichment depends on stale asset data, or the organisation lacks ownership for pipeline rules and retention tiers.

Common Variations and Edge Cases

Tighter telemetry filtering often reduces storage and analyst fatigue, but it also increases the risk of suppressing signals that later prove important, so organisations must balance cost control against forensic completeness.

There is no universal standard for exactly where suppression should occur. In high-risk environments, best practice is evolving toward keeping raw security data in lower-cost storage even when it is excluded from the SIEM, because investigations, legal holds, and incident reconstruction may need the original event later. In regulated sectors, that decision should be mapped to retention and monitoring obligations rather than left to engineering preference alone. The operational question is not whether to collect less, but where each class of telemetry belongs.

Identity-heavy environments also need special handling. Authentication failures, privileged actions, token use, and service account activity often deserve different routing logic from generic infrastructure logs because they are more likely to signal misuse or compromise. The same is true for agentic AI systems that act on behalf of users or services. If those actions are not clearly attributed, pipelines may improve volume management while degrading accountability.

For organisations handling cloud-native or distributed workloads, the model can fail when telemetry arrives too late, in too many formats, or without reliable metadata. In those environments, the pipeline becomes another fragile dependency unless schema governance and detection ownership are explicitly assigned.

Related resources from NHI Mgmt Group

• How do organisations decide where AI data security controls should sit?
• How should security teams replace static secrets in Databricks pipelines?
• How can organisations tell whether their data security programme is actually improving?
• How should organisations evaluate managed services for data security maturity?