Security data pipelines need flexibility without operational lock-in

At a glance

What this is: The article argues that modern security data pipelines need both flexibility and operational control, and that low-code, open-core design is meant to reduce the complexity tax of routing and enrichment.

Why it matters: This matters to IAM and broader security teams because telemetry pipelines increasingly determine what identity, NHI, and threat data is visible, usable, and cost-effective at scale.

👉 Read TENZIR's analysis of low-code security data pipelines and open-core control

Context

Security data pipelines now sit between raw telemetry and the platforms that teams depend on for detection, investigation, and cost control. In practical terms, that means the pipeline has become a governance layer, not just an integration layer, because choices about parsing, enrichment, filtering, and routing shape what the security stack can actually see.

The article's identity angle is indirect but real: if logs, API events, and access telemetry are hard to transform and retain, IAM and NHI programmes lose the evidence they need for review, investigation, and control validation. That is why pipeline design increasingly affects access governance, especially where service accounts, API keys, and machine identities are involved.

Key questions

Q: How should security teams govern multi-tenant telemetry without duplicating pipelines?

A: Security teams should separate tenant execution from central policy oversight. Give each tenant isolated routing, masking, and retention rules, but keep lineage, drift detection, and health monitoring in one control plane. That reduces duplicated engineering work and makes compliance enforceable without rebuilding the same telemetry stack for every customer.

Q: When does telemetry sampling create more risk than it reduces?

A: Sampling creates more risk whenever the stream carries incident evidence, high-value errors, or security-relevant state changes. It is suitable for repetitive success traffic, but not for records that drive detection, auditing, or root-cause analysis. If losing three quarters of the data would change the conclusion, do not sample it.

Q: What do organisations get wrong about low-code security data tools?

A: They often assume low-code means low governance overhead. In practice, even simple operators can create hidden dependencies if no one documents the logic, validates the transformations, or owns lifecycle changes. The real test is whether analysts can understand and safely modify the pipeline without escalating every change to a specialist.

Q: What is the difference between pipeline flexibility and pipeline governability?

A: Flexibility is the ability to build many kinds of data flows. Governability is the ability to understand, review, test, and safely change those flows over time. A platform can be highly flexible but still hard to govern if its logic lives in opaque menus or specialist scripts that only a few people can maintain.

Technical breakdown

Low-code data pipelines versus pro-code complexity

Security data pipelines sit upstream of SIEM, data lakes, and analytics engines, so they need to transform data before expensive back-end processing. The article contrasts GUI-heavy, script-dependent platforms with a pipe-based, low-code model where operators handle parse, enrich, route, and normalise steps in sequence. The core technical issue is not simplicity alone, but whether the pipeline remains readable, versionable, and adaptable when log formats, enrichment logic, and routing rules change. A pipeline that only specialists can safely edit becomes a control bottleneck as well as an engineering one.

Practical implication: treat pipeline logic as governed code, with review, version control, and change accountability.

Open-core security data platforms and control transparency

An open-core model exposes a shared engine and layers enterprise capabilities on top, which changes how teams evaluate trust and extensibility. In security pipelines, transparency matters because parsing, filtering, and routing logic directly influence what evidence survives into downstream tools. Open source does not eliminate operational risk, but it does make the data path more inspectable and can reduce dependence on opaque implementation details. For teams handling security telemetry at scale, the question becomes whether they can audit behaviour, extend the engine, and avoid being trapped by closed vendor abstractions.

Practical implication: validate whether the data path is inspectable enough to support audit, troubleshooting, and long-term portability.

Cost control depends on selective telemetry shaping

The article's SIEM-cost argument reflects a broader architectural pattern: ingest everything and you pay for it, but discard too much and you lose detection fidelity. Selective shaping means aggregating repetitive events, removing low-value noise, and preserving only the telemetry needed for investigation or compliance. Technically, this is a balancing act between reduction and fidelity, and it only works if the pipeline logic is explicit enough to be tested. The design problem is not just data volume reduction, but preserving the identity and security signals that matter most downstream.

Practical implication: define which logs are eligible for reduction and which identity or access events must be preserved intact.

NHI Mgmt Group analysis

Security data pipelines have become governance infrastructure, not plumbing. Once routing and enrichment decide which identity and security events survive into downstream platforms, the pipeline itself becomes part of control assurance. That matters for IAM, NHI, and investigations because missing telemetry is a governance failure, not just an ingestion issue. Practitioners should therefore treat pipeline design as a formal control surface.

Operational complexity is now a security risk multiplier. If only a few specialists can change parsers, filters, or routing rules, the organisation creates a brittle dependency that slows response and increases error risk. This is especially relevant where machine identities generate high-volume telemetry that needs consistent shaping for review. The practical conclusion is that the pipeline must be understandable enough to be governed by the team that depends on it.

Open-core architecture is a transparency position, not a default win. Source visibility can help with trust, extensibility, and long-term portability, but it does not remove the need for access control, secure change management, or validation of custom logic. For identity programmes, the more important question is whether the platform exposes enough of the data path to support audit and lifecycle control. Teams should judge the model by governability, not by ideology.

Telemetry cost pressure is pushing security teams to make harder architectural choices. The decision is increasingly between paying for broad ingest or shaping data closer to the source with explicit policy. That creates a new linkage between security economics and evidence quality, especially in identity-heavy environments where access, token, and service-account events must remain usable. Practitioners should expect pipeline governance to become part of security architecture reviews.

Pipeline extensibility now matters because security environments change faster than vendor assumptions. New log sources, new cloud services, and new machine-identity patterns all create parsing and routing demands that closed workflows may not absorb cleanly. The named concept here is pipeline control debt: the growing gap between what a pipeline must do and how many people can safely change it. Teams should minimise that debt before it becomes operational drag.

What this signals

Security teams should expect the telemetry layer to be treated as part of the control plane, especially when identity and machine-identity events determine whether investigations can be reconstructed. The most useful architectural shift is to make pipeline logic reviewable and portable before the organisation becomes dependent on brittle, specialist-only workflows.

Pipeline control debt: when transformation logic becomes too complex for the wider team to maintain, operational risk rises faster than tooling flexibility. This is especially relevant in environments where service-account, API-key, and access telemetry must remain intact for IAM and NHI oversight.

For practitioners

• Version-control pipeline logic Store parsers, routing rules, and enrichment logic in Git so changes are reviewable, testable, and rollback-ready rather than trapped in a GUI.
• Define telemetry preservation rules Identify identity, access, and high-signal security events that must never be reduced or dropped, then codify those preservation rules before cost optimisation begins.
• Audit specialist dependency points Map which pipeline components require deep product expertise and reduce single-owner bottlenecks through documentation, peer review, and operational runbooks.
• Test portability before lock-in hardens Check whether parsing, enrichment, and output logic can be moved across platforms without rewriting the control model, especially for SIEM and data-lake destinations.

Key takeaways

• Security data pipelines are now a governance layer because they determine which telemetry reaches downstream detection and analytics systems.
• The article's central trade-off is between flexibility and maintainability, with specialist-only pipeline logic creating operational fragility.
• Practitioners should manage pipeline logic as controlled code, preserving high-value identity and access events while reducing noise selectively.

Key terms

• Security data pipeline: A security data pipeline is the chain that ingests, filters, enriches, normalises, and routes telemetry before it reaches storage or analytics. In practice, it determines which evidence survives into detection, investigation, and compliance workflows, so it is part of the control environment, not just infrastructure plumbing.
• Telemetry Shaping: Telemetry shaping is the deliberate reduction, aggregation, or transformation of event data before storage or analysis. Done well, it removes noise without destroying investigative value. Done poorly, it creates blind spots by stripping context from identity, access, or attack signals.
• Open core software: A model where the core product code is openly available while commercial capabilities, support, or hosting may sit around it. In identity infrastructure, this can improve adoption, but it also creates governance questions about continuity, compatibility, and how production trust is maintained over time.

What's in the full article

TENZIR's full article covers the operational detail this post intentionally leaves for the source:

• The TQL operator examples that show how routing, enrichment, and shaping are chained in practice
• The product comparison scenarios that contrast parser handling, cost reduction, and platform lock-in
• The open-core architecture argument, including how the underlying engine supports extension and transparency
• The community library workflow for reusable parsers, pipelines, and context packages

👉 TENZIR's full article covers the pipeline examples, comparison scenarios, and open-core rationale in more detail.

Deepen your knowledge

NHI Mgmt Group's NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to the broader security architecture their programme depends on.