Security data pipelines expose the governance gap in SIEM-centric SOCs

At a glance

What this is: DataBahn argues that SIEM-centred security data pipelines create a governance gap around cost, ingestion resilience, and schema handling.

Why it matters: IAM, NHI, and SOC teams need to treat telemetry routing and enrichment as control decisions because ingestion design now affects loss, lock-in, and operating cost.

By the numbers:

• ~70% of the data being sent to SIEMs is not security-relevant.
• DataBahn says it delivers a 35%+ reduction in SIEM costs by reducing log sizes in 2-4 weeks.

Context

Security data pipelines are the control layer that decides how telemetry is collected, transformed, routed, and retained before it reaches a SIEM or other downstream destination. When that layer is treated as plumbing only, organisations end up paying for volume they do not need, absorbing schema conversion work repeatedly, and accepting pipeline fragility as normal.

The identity connection is real even in a broad SOC article: service accounts, API keys, connectors, and agent credentials are what make these pipelines work, and those credentials need lifecycle governance, scoping, and offboarding just like any other privileged access path. In that sense, the issue is not only log transport but the governance of machine access to telemetry flows.

Key questions

Q: What breaks when security data pipelines are treated as simple plumbing?

A: The control model breaks because routing, enrichment, and retention decisions get made too late. That leads to higher SIEM costs, more noise, and a greater chance of blind spots when pipelines fail or data volumes spike. Security teams should treat the pipeline as part of the detection control surface, not just as transport.

Q: Why do SIEM-centric pipelines create hidden governance risk?

A: Because the organisation pays for ingestion volume even when much of the data has little security value. That creates pressure to centralise everything, which increases cost, lock-in, and operational dependency on a fragile path. Governance has to decide what is worth retaining at full fidelity and what is not.

Q: What are the signs that telemetry enrichment is failing in practice?

A: Enrichment is likely failing when data cannot be separated by the attributes teams expect, when troubleshooting still requires manual searching, or when logs and metrics remain hard to classify. Another warning sign is that attributes appear inconsistently across sources or destinations, which usually means the processor placement or configuration is not aligned with the intended data flow.

Q: How should security teams govern machine identities in industrial environments?

A: Security teams should govern machine identities the same way they govern privileged access: assign an owner, define a specific purpose, limit scope, and review it continuously. In practice, that means tracking service accounts, certificates, APIs, and connectors as non-human identities with their own lifecycle, not as background infrastructure. A machine identity should never have broader access than its workflow requires.

Technical breakdown

Why SIEM-centric ingestion becomes a governance bottleneck

A SIEM-centric model treats the SIEM as the primary destination for raw telemetry, then expects engineering teams to solve noise reduction, format variance, and routing after the fact. That approach works until the volume, diversity, or cost profile of data exceeds what the SIEM can economically absorb. Once that happens, ingestion policy becomes a governance decision because every byte forwarded has budget, retention, and detection trade-offs attached. The problem is not the SIEM alone. It is the assumption that ingestion can remain a downstream, centralised function while sources and use cases keep multiplying.

Practical implication: Move ingestion policy earlier in the stack and define which telemetry classes deserve full-fidelity SIEM treatment.

How pre-SIEM enrichment changes security data handling

Pre-SIEM enrichment attaches context before the routing decision is made, not after storage has already been paid for. That can include asset identity, threat intelligence, geolocation, or environment classification. Once the event is enriched, the pipeline can decide whether it merits full retention, lower-cost storage, or further transformation. This changes the control logic from volume-first to signal-first. The key architectural point is that enrichment and filtering are inseparable: filtering without context is blind, while enrichment before ingestion makes the routing decision defensible.

Practical implication: Place enrichment where it can influence routing, retention, and prioritisation before SIEM licensing costs are incurred.

Why pipeline resilience matters as much as log content

A telemetry pipeline is not just a transport path. It is a dependency chain that can fail under back pressure, volume spikes, schema changes, or connector issues. If that chain is single-path, organisations can lose data before it ever reaches the SOC. Resilient designs use secondary channels, buffering, observability, and transformation controls so ingestion continues even when one path degrades. In governance terms, this is about avoiding silent telemetry loss, not merely improving throughput. If the pipeline is the only path to visibility, its failure becomes a security failure.

Practical implication: Design for loss-tolerant telemetry ingestion with failover paths and pipeline health monitoring.

Threat narrative

Attacker objective: The practical objective is to reduce or blind security visibility by exploiting weak ingestion governance and pipeline fragility.

1. Entry occurs through telemetry sources and connectors that must authenticate into the pipeline, which makes machine credentials and integration trust part of the attack surface.
2. Escalation happens when volume spikes, schema drift, or back pressure overwhelm a fragile pipeline and create conditions for dropped or delayed events.
3. Impact is loss of visibility, delayed SOC detection, and higher operating cost because the organisation either over-ingests everything or accepts blind spots.

Breaches seen in the wild

• CI/CD pipeline exploitation case study: Credentials in an exposed .git/config let a researcher edit a Bitbucket pipeline so it planted their SSH key on the server. No victim was named.
• reviewdog Action compromise 2025: A stolen maintainer token poisoned reviewdog/action-setup, leaking CI secrets including the tj-actions bot token used in the next attack.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Security data pipeline governance is now an identity problem as much as a data engineering problem. Every collector, connector, and routing service depends on machine credentials that need ownership, scope, and offboarding. That makes telemetry infrastructure part of IAM and PAM, not a separate engineering concern. Practitioners should treat pipeline access as governed machine identity.

SIEM-centred architectures create governance debt when they assume every event deserves equal treatment. The article's core numbers show why that assumption breaks down: high ingest waste, inflated storage, and weeks of integration effort for new sources. When a control model rewards volume rather than relevance, organisations pay twice, once in licence cost and again in analyst noise. Practitioners should re-evaluate where enrichment and triage decisions sit.

Loss-less ingestion is the real resilience requirement, not faster forwarding. Back pressure and spikes are not just performance issues because they can erase evidence before it reaches the SOC. That shifts the control question from throughput tuning to continuity of visibility. Practitioners should map ingestion resilience to business-critical detection use cases and design secondary paths accordingly.

Schema translation has become a hidden governance layer in security operations. Proprietary formats, custom apps, and destination-specific schemas force manual transformation work that is usually invisible in budget discussions. The result is vendor lock-in by operational friction, not by contract alone. Practitioners should treat normalisation rules and transformation ownership as governed controls, not incidental plumbing.

Security data fabrics are a response to SOC over-centralisation, but they only help if the operating model changes with them. Moving enrichment upstream, routing by value, and separating collection from analysis can reduce cost and improve fidelity, but only when teams also redefine accountability for data quality and pipeline health. Practitioners should align governance with the architecture shift, or the old bottlenecks will simply reappear elsewhere.

What this signals

Security data pipelines now sit at the intersection of cost governance and identity governance. The more sources an SOC ingests, the more machine credentials, schemas, and routing rules it has to govern. That means pipeline architecture should be reviewed alongside IAM and PAM, not after the fact. Practitioners should expect telemetry controls to become part of their broader identity fabric.

Telemetry enrichment only helps when it happens before retention decisions are locked in. Once raw data has already been committed to SIEM storage, enrichment becomes forensic context rather than a control that changes outcomes. The programme lesson is to move classification and routing left, then measure whether the pipeline actually reduces noise without creating blind spots.

For practitioners

• Define pipeline ownership as a governed control Assign accountable owners for collectors, connectors, transformations, and routing rules so machine access to telemetry flows is tracked like any other privileged dependency.
• Classify telemetry by security value before SIEM ingestion Create routing policies that distinguish high-value security events from routine data so full-fidelity SIEM storage is reserved for detections that justify the cost.
• Reduce brittle integration paths for new sources Standardise onboarding patterns for custom apps and unstructured sources so adding telemetry does not require repeated manual engineering work or long-lived exceptions.
• Build failover into ingestion paths Use secondary channels, buffering, and pipeline health monitoring so spikes or back pressure do not turn into silent data loss.
• Treat schema normalisation as a control, not a cleanup task Document who owns data transformation, enrichment, and normalisation so proprietary formats do not create hidden lock-in or inconsistent security coverage.

Key takeaways

• Security data pipeline design is a governance issue because it determines where cost, loss, and visibility risks are absorbed.
• The article shows that SIEMs can become expensive and brittle when they are forced to accept raw telemetry before enrichment and routing.
• Practitioners should govern pipeline ownership, resilience, and machine identity lifecycle before expanding ingestion scope.

Key terms

• Security data pipeline: A security data pipeline is the chain that ingests, filters, enriches, normalises, and routes telemetry before it reaches storage or analytics. In practice, it determines which evidence survives into detection, investigation, and compliance workflows, so it is part of the control environment, not just infrastructure plumbing.
• Pre-SIEM Enrichment: Pre-SIEM enrichment is the process of attaching security context to telemetry before it reaches the SIEM. That context can include identity data, asset ownership, geolocation, or threat intelligence, allowing teams to make a routing decision before they pay indexed-storage costs.
• Loss-less Ingestion: Loss-less ingestion is the goal of receiving and forwarding telemetry without dropping events when volume rises or a path becomes blocked. It usually requires buffering, alternate routes, and health checks so security evidence remains complete enough for detection and investigation.
• Schema Normalization: Schema normalization is the process of converting inconsistent raw log fields into a stable structure that downstream systems can reliably use. It reduces parsing drift, improves correlation accuracy, and prevents each tool from having to solve vendor-specific formatting problems on its own.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to govern privileged access across pipelines, platforms, and production systems.