Security data pipeline platforms expose the limits of legacy SIEMs

At a glance

What this is: Security data pipeline platforms are emerging because legacy SIEMs struggle to absorb modern telemetry volume without degrading cost, routing quality and analytic performance.

Why it matters: IAM, NHI and security teams need to govern how telemetry is enriched, filtered and routed before ingestion, because the control point is shifting upstream from the SIEM itself.

Context

Security operations now depend on data quality as much as detection logic. When telemetry arrives late, noisy or over-retained, SIEMs become expensive storage and brittle analytics systems instead of decision engines. The primary issue is not just volume, but whether the pipeline can preserve signal while controlling what reaches high-cost analysis layers.

This is a governance problem with an identity dimension because routing, enrichment and correlation increasingly depend on trustworthy machine and human context. If service accounts, API-driven collectors or agentic workflows feed the pipeline, the organisation also has to know which identities are allowed to move, transform and prioritise security data.

Key questions

Q: How should security teams decide which telemetry belongs in the SIEM?

A: Start with investigative value, not source count. High-fidelity SIEM retention should be reserved for telemetry that materially improves detection, forensics, or compliance. Lower-value data can be enriched first, routed to cheaper storage, or dropped if it adds cost without operational benefit. The decision should be policy-driven, measurable, and reviewed against detection outcomes.

Q: Why do legacy SIEMs struggle when telemetry volume keeps rising?

A: Because storage, search, and correlation costs rise faster than the quality of the signal. Once teams ingest everything first and decide later, the SIEM becomes a bottleneck for performance, budget, and analyst time. The result is often delayed detection, selective visibility, and too much effort spent managing pipelines instead of threats.

Q: What breaks when enrichment happens only after SIEM ingestion?

A: Three things usually break together: cost control, detection speed, and retention discipline. The organisation has already paid ingest pricing, analysts still need to add context manually, and noisy data competes with high-value telemetry in the same storage tier. Once the event is stored, the chance to make a smarter routing decision has passed.

Q: How should security teams govern machine identities in industrial environments?

A: Security teams should govern machine identities the same way they govern privileged access: assign an owner, define a specific purpose, limit scope, and review it continuously. In practice, that means tracking service accounts, certificates, APIs, and connectors as non-human identities with their own lifecycle, not as background infrastructure. A machine identity should never have broader access than its workflow requires.

Technical breakdown

Why SIEM ingestion breaks under telemetry scale

Legacy SIEM architectures assume most incoming data can be normalised, stored and queried centrally without materially changing throughput or cost. That assumption fails when hundreds of sources generate terabytes of telemetry each day. The problem is not only storage pressure. Every enrichment lookup, parsing step and correlation rule adds latency, and the SIEM becomes the choke point for both security operations and budget control. Once ingestion cost drives filtering decisions, teams stop asking what to detect and start asking what they can afford to keep.

Practical implication: move high-volume filtering and enrichment earlier in the pipeline so the SIEM receives only signal-bearing events.

How security data pipeline platforms change routing and enrichment

Security data pipeline platforms decouple collection from storage and analysis. That means telemetry can be enriched, classified and routed before it is committed to an expensive downstream system. In practice, this turns routing into a policy decision based on context such as asset identity, threat indicators and event value. The architectural value is not that data moves faster, but that the organisation can preserve evidence while reducing unnecessary ingestion. This is especially relevant where machine identities, collectors and AI-driven automations now generate or transform telemetry at scale.

Practical implication: define routing policy for enriched telemetry, not raw logs, so value-based retention becomes repeatable.

Why AI in the pipeline is not just automation

AI in this context should not be treated as a query shortcut. The more useful pattern is intelligence embedded across collection, transformation and insight generation so the pipeline can adapt to changing formats and event patterns. That matters because manual detection engineering cannot keep pace with source sprawl. For identity-heavy environments, the same logic applies to machine identities and service accounts that generate telemetry, where governance needs to operate at the point of event creation and transformation, not only in the SIEM or SOC console.

Practical implication: evaluate whether AI is reducing analyst friction across the full data lifecycle or only masking ingestion inefficiency.

Threat narrative

Attacker objective: The operational objective is to reduce the defender's ability to see, prioritise and act on meaningful security events in time.

1. Entry occurs through the data layer itself, where a brittle or overloaded SIEM path cannot distinguish useful telemetry from noise at scale.
2. Escalation happens when enrichment, parsing and routing slow down enough that teams start dropping data, over-retaining everything, or depending on manual triage.
3. Impact is reduced detection fidelity, higher SIEM cost and weaker response because analysts spend time managing the pipeline instead of investigating threats.

Breaches seen in the wild

• CI/CD pipeline exploitation case study: Credentials in an exposed .git/config let a researcher edit a Bitbucket pipeline so it planted their SSH key on the server. No victim was named.
• Shai Hulud npm malware campaign: Shai Hulud campaign: npm malware exposed secrets on GitHub.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Security data pipelines are becoming a governance layer, not a plumbing layer. Once telemetry volume, enrichment and routing determine which events reach SIEM retention, the control point moves above the detection engine. That means security leaders are now deciding how signal is preserved, not just where logs are stored. For practitioners, the key issue is pipeline governance, not SIEM tuning.

Telemetry trust now depends on the identities that move the data. Where collectors, connectors, service accounts or agentic workflows transform events before ingestion, those identities become part of the control surface. That creates an NHI governance problem inside a cloud or SOC architecture, because the wrong machine identity can distort what gets enriched, routed or dropped. Practitioners should treat telemetry-moving identities as privileged.

Cost optimisation is no longer separable from detection quality. The article shows that filtering and enrichment before SIEM ingestion are really the same policy decision made with better context. That matters because organisations still often treat storage economics and detection engineering as separate teams. In practice, the strongest control is to make retention tiering dependent on enriched security value, not raw event count.

Agentic AI in the pipeline increases the need for bounded automation. If AI is used to build connectors, resolve failures or prioritise events, it must be governed as a non-human decision path with clear limits and accountability. The security issue is not automation itself but whether the pipeline can explain why a machine identity changed routing, enrichment or escalation behaviour. Practitioners should require auditable AI actions in data engineering workflows.

Data volume is exposing a detection architecture gap that older SIEM models were never meant to close. The market is moving toward control planes that sit beneath SIEM and shape what the SIEM actually sees. That validates the need for upstream governance of event quality, source trust and routing policy. Practitioners should expect detection to become more distributed across the pipeline rather than concentrated in one platform.

What this signals

Security data pipeline platforms are pushing detection control upstream. The practical consequence is that SOC teams will increasingly need policies for telemetry quality, event prioritisation and routing before the SIEM ever sees the data. That changes programme design because the security team now governs a data control plane, not just a logging backend.

Machine identities are part of the telemetry trust model. Connectors, collectors and AI-assisted pipeline actions can all change what evidence survives into downstream analytics. For IAM and NHI practitioners, that means pipeline accounts need the same lifecycle discipline and privilege review expected of other high-risk service identities.

Detection economics and governance are converging. When expensive SIEM ingestion is reserved for enriched, high-value events, storage reduction becomes a by-product of better security control. Teams that still treat cost optimisation and detection engineering as separate programmes will miss the real architectural change.

For practitioners

• Define telemetry routing policy by event value Classify events before SIEM ingestion so high-value telemetry receives full-fidelity retention and low-value telemetry is routed to cheaper storage.
• Inventory the identities that move security data Document the service accounts, API keys and connector credentials that collect, enrich and forward telemetry so you can govern them as privileged machine identities.
• Separate enrichment from synchronous ingestion Avoid line-in-the-path external lookups that slow or block event flow, and use pre-indexed or asynchronous enrichment where possible.
• Measure SIEM value by signal, not volume Track how much telemetry reaches expensive retention tiers because it supports detections, investigations or compliance needs rather than because it was easiest to keep.

Key takeaways

• Legacy SIEM architectures struggle when telemetry volume, enrichment and routing overhead exceed what central ingestion was designed to handle.
• The practical response is to move context-aware filtering and routing upstream so the SIEM receives fewer, higher-value events.
• Machine identities that run collectors, connectors and automations should be governed as privileged parts of the telemetry control plane.

Key terms

• Security data pipeline: A security data pipeline is the chain that ingests, filters, enriches, normalises, and routes telemetry before it reaches storage or analytics. In practice, it determines which evidence survives into detection, investigation, and compliance workflows, so it is part of the control environment, not just infrastructure plumbing.
• Telemetry Enrichment: Telemetry enrichment is the process of adding context to raw security data so it is more useful for investigation and response. That can include asset, identity, geolocation, or threat intelligence context, but enrichment must be controlled so it does not distort the original evidence record.
• Pre-SIEM Routing: Pre-SIEM routing is the practice of deciding where telemetry should go before it reaches the expensive SIEM tier. Enriched context is used to separate high-value events from routine ones, allowing teams to retain useful records while controlling ingestion, storage, and query cost.
• Non-Human Identity (NHI): A digital identity assigned to a non-human entity such as a software application, service account, API key, bot, machine, or AI agent that enables it to authenticate and interact with systems without direct human involvement. NHIs now outnumber human identities in most enterprises by 25 to 50 times.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity and secrets management for teams that need tighter control over machine identities. It helps identity and security practitioners apply governance discipline to service accounts, connectors and other non-human controls that support modern security operations.