Security Data Pipeline Buyer's Guide

Security data pipeline platforms are usually bought to cut SIEM ingest bills, and a platform chosen on reduction percentages alone can quietly drop the events your detections depend on. The category sits between log sources and destinations: it collects, parses, normalises, enriches, filters and routes telemetry to a SIEM, a security data lake, cheap archive storage and observability tools. This vendor-neutral guide helps security operations leads, detection engineers, platform teams and procurement choose a security data pipeline platform: which category to buy, requirements, evaluation criteria, pricing models, RFP questions, red flags and a proof of concept. For how pipelines work, see the Security Data Pipeline Guide.

Key takeaways

Buy for routing and control, not just reduction. The durable value is choosing where each event goes, in what format, with what guarantees; volume savings follow from good routing.
Detection engineers must approve every reduction rule. The CISA and ASD event logging guidance supports filtering before the SIEM to save cost, but a dropped field can silently break a rule.
Open schemas and open protocols limit lock-in. Prefer platforms that output OCSF or OpenTelemetry semantic conventions, accept OTLP and syslog over TLS, and export their configuration as code.
Delivery guarantees matter more than throughput. Ask how data survives a destination outage: persistent queues, acknowledgements, retry and replay.
The pipeline is a high-value target. It holds credentials to every log source and destination and can blind detection if tampered with (ATT&CK T1685); secure it as tier zero.

Define requirements

Choosing a security data pipeline starts with these decisions, written down before vendor meetings.

Outcomes: lower SIEM ingest cost, a SIEM migration, feeding a security data lake, sending one stream to several teams, or meeting retention rules cheaply. Rank them, because a migration-led purchase weighs format conversion and replay; a cost-led purchase weighs reduction and tiering.
Sources and volumes: list sources with daily volume, peak rate, protocol and format: endpoint and EDR, identity provider and directory, cloud audit logs, firewall and edge devices, DNS, proxy and SaaS audit APIs. The CISA and ASD guidance prioritises edge devices such as "boundary routers and firewalls" and cloud logs "for cloud APIs, all network-related events".
Destinations: current and planned SIEM, a security data lake, object storage archive, observability back end and any regulator or customer feeds. See the Security Data Lake Guide.
Schema strategy: decide the internal contract, such as OCSF for security events or OpenTelemetry semantic conventions for shared telemetry, before vendors propose their own.
Retention and residency: how long raw data must be kept and where, and whether a hosted control plane may see event content. The guidance warns that "Default log retention periods are often insufficient."
Operating model: who owns the pipeline (security engineering, a platform team, observability), and whether you can run open-source collectors as a product or need a managed service.

Size changes the answer. A small organisation with one SIEM and modest volume usually does best with the SIEM's or XDR's own collectors and no separate pipeline at all. A pipeline earns its place when there are several destinations, a planned SIEM migration, ingest costs above what the business will fund, or a security data fabric strategy in which security, IT operations and observability share one collection layer.

Capability areas and evaluation criteria

Several overlapping labels describe this market. A fair security data pipeline comparison puts like with like.

Label	What it usually means	Fit for security
Security data pipeline platform	Pipeline with security parsers, OCSF mapping, SIEM and lake destinations and reduction packs	Built for this use case; check observability support if shared
Telemetry pipeline platform or observability pipeline	Pipeline for logs, metrics and traces from an observability heritage, often OpenTelemetry-native	Strong throughput and OTLP; check security parsers, delivery guarantees and audit sources
Log pipeline (agent and aggregator)	Collector and forwarder software, open source or commercial, configured as code	Low licence cost; you build parsers, management and monitoring
Security data fabric	Marketing term for a pipeline plus a management layer spanning many sources, destinations and teams	Ask what is actually delivered beyond routing and a catalogue
Data pipeline management platform	Control plane that manages fleets of collectors and pipeline workers	Useful at scale; separate the control plane's access from the data plane
SIEM-native ingestion	The SIEM vendor's collectors, parsers and ingest-time rules	Simplest with one destination; weak for multi-destination and migration

The OpenTelemetry Collector illustrates the open baseline that commercial telemetry pipeline vendors and log pipeline vendors build on or compete with: the project describes it as "a vendor-agnostic implementation of how to receive, process and export telemetry data", and OpenTelemetry moved to the CNCF Graduated maturity level on 11 May 2026. OCSF is "agnostic to storage format, data collection and ETL processes" and licensed under Apache 2.0, so a pipeline that maps to it leaves your data portable.

Weight criteria to your outcomes. An illustrative split: 20% collection and delivery, 20% parsing and normalisation, 20% reduction and routing with detection safety, 15% pipeline security, 15% operations and portability, 10% commercial terms.

Criterion	What good looks like	Weak signs
Collection	Agents and agentless pull for your sources; syslog over TLS (RFC 5425); OTLP on 4317 (gRPC) and 4318 (HTTP); cloud and SaaS audit APIs; Windows event forwarding	Source list padded with generic HTTP inputs
Delivery guarantees	Disk-backed persistent queues, end-to-end acknowledgements, back-pressure, honours OTLP retryable codes (429, 502, 503, 504) and partial success; dead-letter handling	Memory-only buffers; silent drops when a destination is slow
Normalisation	Maintained parsers; OCSF output with class_uid, category_uid and activity_id populated; unparsed fields kept in unmapped and the original in raw_data	Proprietary schema only; fields discarded when unmapped
Timestamps and identifiers	Original event time preserved, UTC and ISO 8601, millisecond precision, unique event ID	Ingest time overwrites event time
Reduction	Field-level trimming, deduplication, aggregation with counts, sampling only for non-security data; every rule previewed against live data and versioned	Headline "cut volume by X%" with no rule-by-rule view
Routing and tiering	Content-based routing to SIEM, lake and archive; full-fidelity copy to cheap storage; replay or rehydration of archived data into the SIEM	Archive in a proprietary format readable only by the vendor
Enrichment	Asset, identity, geolocation and threat intelligence lookups with cache and refresh controls	Enrichment that blocks the pipeline when a lookup fails
Sensitive data	Redaction and masking rules, hashing of identifiers, per-destination policies	No way to keep PII out of a third-party destination
Operations	Configuration as code in Git, staged rollout, rollback, pipeline health metrics (events in, out, dropped, queued, lag)	Click-only configuration; no drop counters
Portability	Exportable configuration, open formats at rest, documented exit	Parsers and rules that exist only in the vendor console

The CISA and ASD guidance recommends "a centralised event logging facility such as a secured data lake" with "select, processed logs" forwarded to the SIEM, "a structured log format, such as JSON", automated normalisation, UTC with ISO 8601 timestamps, and filtering before the SIEM "to minimise any additional costs or capacity issues". It also warns that SIEMs "can be a single point of failure"; the same is true of a pipeline in front of them.

Pricing models

Security data pipeline pricing differs enough that the same estate can cost several times more under one model than another. Model your volumes, not the vendor's example.

Model	What is counted	Watch for
Per volume ingested or processed	GB or TB per day entering the pipeline	Paying on pre-reduction volume, so savings land in the SIEM bill but not the pipeline bill
Per node, worker or core	Pipeline compute you run or the vendor hosts	Peak sizing for bursts; high-availability pairs counted twice
Per source or agent	Hosts with agents or connected sources	Ephemeral workloads and containers inflating counts
Consumption commit	Pre-paid credits drawn down by volume or features	Overage rates; expiry of unused credits
Open source plus support	Support subscription, enterprise features or management plane	Engineering time to build and maintain parsers and monitoring

Add cloud egress and cross-region transfer, archive storage, rehydration charges and professional services, and ask for a three-year cost at today's volume and at 50% growth.

Where identity fits

A pipeline holds some of the most privileged non-human identities in the estate: read access to every cloud audit log, API tokens for SaaS audit endpoints, and write tokens for the SIEM and lake. Require workload identity federation or role assumption instead of static cloud keys, secrets held in a vault rather than configuration files (the OpenTelemetry guidance says to store them "on an encrypted filesystem or secret store"), separate credentials per source and destination, and alerting on their use from unexpected locations. When a 2023 breach at a log analytics provider involved a compromised AWS credential, customers were left rotating the credentials and keys they had given the service. See the Service Account Security Guide.

Framework mapping

Framework	Reference	What the platform should evidence
NIST SP 800-53 Rev. 5	AU-3 Content of Audit Records; AU-8 Time Stamps	Fields preserved through parsing; original event time in UTC
NIST SP 800-53 Rev. 5	AU-4 Audit Log Storage Capacity; AU-11 Audit Record Retention	Persistent queues; full-fidelity archive tier
NIST SP 800-53 Rev. 5	AU-9 Protection of Audit Information	TLS in transit, integrity checks, restricted pipeline administration
NIST SP 800-53 Rev. 5	AU-6 Audit Record Review; SI-4 System Monitoring	Reliable delivery to SIEM; enrichment for analysis
NIST SP 800-92 Rev. 1 (draft)	Log management playbook: generating, transmitting, storing, accessing and disposing of log data	Pipeline design covers each stage
MITRE ATT&CK	T1685 Disable or Modify Tools (including T1685.001 and T1685.002)	Alerts on silent sources, configuration changes and drop spikes

Questions to ask vendors

Use these as the core of a security data pipeline RFP and ask security data pipeline vendors, and any observability pipeline vendors on the shortlist, the same set in writing, with answers shown on your data in the proof of concept.

Which of our sources do you collect natively, by which protocol, and which parsers are maintained by you versus the community or us?
How does data survive a SIEM outage of four hours: where is it buffered, how much disk is needed, and how is replay ordered?
Do you output OCSF? Which version, which classes for our sources, and what happens to fields you cannot map?
Show every reduction rule you propose for our authentication, DNS and firewall data, the fields removed and the detections that use them. Can we preview and version each rule?
Can we keep a full-fidelity copy in our own storage in an open format, and rehydrate a time range into the SIEM? What does rehydration cost?
How are pipeline credentials stored, can sources and destinations use federation instead of static keys, and what can your staff access in our data plane?
Is the control plane hosted? What event content, if any, leaves our environment, and where is it processed?
How do we detect a source going silent, a pipeline configuration change or a spike in drops, and can those alerts go to our SIEM?
Can configuration live in Git with staged rollout and rollback? What does the exit process look like, and in what format do we get our pipelines back?
What exactly counts toward the licence, on pre- or post-reduction volume, and what does 50% growth cost?

Red flags

Reduction claims quoted as a percentage with no rule-level review, or a default pack that drops failed authentication or DNS events.
Memory-only buffering for security sources.
Parsers and routing logic that cannot be exported, so leaving the platform means rebuilding every pipeline.
Archive stored in a proprietary compressed format that only the vendor's tooling can read.
Collectors listening on all interfaces without authentication; the OpenTelemetry guidance says to "always use specific interfaces" rather than 0.0.0.0 and to avoid running the collector as root.
"AI-generated parsers" with no test results against your samples.
Licensing on pre-reduction volume that captures most of the promised SIEM saving.

Proof of concept

Pick three to five sources: include one high-volume source (firewall or DNS), one identity source (directory or identity provider sign-ins) and one cloud audit source, with real daily volumes.
Baseline: record current ingest per source, SIEM cost per day and a list of detections using each source, with their test cases.
Dual-run: send raw data down the existing path and through the candidate in parallel to a test SIEM index and to object storage, so you can compare event counts and fields.
Validate detections: replay test cases (password spraying, impossible travel, new admin role assignment, DNS to a known bad domain) and confirm every detection still fires after normalisation and reduction.
Break things: stop the SIEM destination for an hour, restart a pipeline node, send malformed events and a burst at three times normal rate. Measure loss, lag and recovery.
Check security: confirm TLS on every hop (the CISA and ASD guidance recommends TLS 1.3), credentials in a vault, collectors bound to specific interfaces and alerts on configuration change.
Roll back cleanly: switch sources back to the original path, revoke pipeline credentials, confirm in source logs that access has stopped and get written confirmation that trial data has been deleted.

Carry these measures into production: events dropped unintentionally (target zero), median end-to-end latency per destination, share of events normalised without unmapped core fields, detections passing regression tests after each pipeline change, ingest cost per source, and time to restore after a destination outage.

Standards and references

How NHI Mgmt Group can help

We provide independent requirements, RFP and evaluation support. Browse vendors in our products directory or contact us.

Security Data Pipeline Buyer’s Guide