Security Data Pipeline Guide

Between every log source and the SIEM sits a pipeline that decides what the security team will ever see, and it is usually the least governed part of the detection stack. Collectors and stream processors parse, enrich, drop and route billions of events a day, and one careless filter or full buffer can remove evidence without raising an alert. This guide covers the stages of a security data pipeline, normalisation schemas (OCSF, the Elastic Common Schema and OpenTelemetry semantic conventions), buffering and delivery guarantees, safe and unsafe data reduction, routing to SIEM, data lake and archive tiers, and how to secure the pipeline and the non-human identities that run it. It is written for detection engineers, platform engineers and SOC leads; open-source projects are used as examples from their documentation, and the practices apply equally to commercial pipeline products.

Key takeaways

• The pipeline is part of your detection logic. Every parse, drop and route decision changes what rules can match, so the configuration needs the same review and testing as detection content.
• Normalise once, keep the raw event. Map to one schema (OCSF, ECS or OpenTelemetry semantic conventions) and keep the original record so mapping mistakes can be corrected.
• Reduce volume by purpose, never by sampling security events. Dropping unused fields, deduplicating and summarising high-volume flows are safe; sampling authentication or audit events silently breaks threshold and correlation rules.
• Design for loss, then measure it. Memory buffers lose data and full disk buffers discard the oldest chunks. Use durable buffers for security sources and alert on discards.
• Treat pipeline credentials as privileged NHIs. Collector service accounts, API keys that pull SaaS audit logs and SIEM ingest tokens can read, alter or suppress all security telemetry; ATT&CK T1685 explicitly covers "interfering with logging pipelines and forwarding mechanisms".

What a security data pipeline does

A security data pipeline (also called a telemetry or observability pipeline) takes events from sources and delivers them, transformed, to one or more destinations. NIST's draft SP 800-92 Rev. 1 defines log management as "the process for generating, transmitting, storing, accessing, and disposing of log data"; the pipeline owns transmission and, increasingly, storage decisions. Decoupling collection from the SIEM lets you change SIEM vendors, feed a data lake and control cost without touching every source. The OpenTelemetry Collector uses receivers, processors and exporters joined into pipelines (with connectors linking two pipelines); Vector uses sources, transforms and sinks; Fluent Bit uses inputs, parsers, filters, a buffer, a router and outputs.

Use cases and source priorities are in the SIEM Guide; this page covers the layer underneath.

Collection, transport and delivery guarantees

Transport

• Syslog: RFC 5425 defines syslog over TLS on TCP port 6514 and says masquerade, modification and disclosure threats "are mitigated only if both the transport sender and transport receiver are properly authenticated": validate certificates on both sides. Retire plain UDP 514 for detection sources.
• OTLP: OpenTelemetry's protocol uses port 4317 for gRPC and 4318 for HTTP by default. Enable TLS and an authenticator on receivers that accept data from outside the host.
• Listening interfaces: the OpenTelemetry security guidance says to use "specific interfaces, such as a pod's IP, or localhost instead of 0.0.0.0"; localhost has been the default since Collector v0.110.0, but older copied configurations still bind everywhere.
• API pull sources: track each SaaS or cloud audit integration's polling lag and API quota; a throttled poller produces gaps that look like quiet periods.

Buffering and acknowledgement

Pipelines lose data in documented ways. Fluent Bit says memory-only buffering is faster "but is more prone to data loss"; when mem_buf_limit is reached the input pauses, the memrb ring buffer mode drops the oldest chunks, and once filesystem storage reaches storage.total_limit_size "the oldest chunk from its queue will be discarded to make room for new data". Vector's end-to-end acknowledgements let a source learn whether events reached their sinks: if an event goes to three sinks and only two succeed it is marked failed, and "the 'worst' status is the status reported to the source". That only helps when the source can acknowledge (a message queue, a retrying HTTP sender); UDP syslog cannot.

• Security sources get durable buffers: disk buffers sized for the longest plausible destination outage, not the default. For audit data, back-pressure into a source that retries beats discarding the oldest events.
• Queue large estates: a durable message bus absorbs bursts and allows replay after a bad transform.
• Alert on discard counters: a non-zero discard count on a security source is an incident for the platform team.

Parsing and normalisation schemas

Normalisation means one field name for one concept, so one query works across vendors. Three open schemas dominate.

• OCSF structure: each event class has a class_uid, belongs to a category through category_uid (categories include System Activity, Network Activity, Identity & Access Management, Findings, Discovery and Application Activity) and carries an activity_id. The type_uid is calculated as class_uid * 100 + activity_id, one filterable value per event type. Profiles (Host, Cloud, Container, Security Control) add attribute sets; metadata.version records the schema version.
• OCSF escape hatches: fields that do not map can go in the optional unmapped attribute, and raw_data "holds the event data as received from the source". Use both, or keep a pointer to the raw record in cheap storage.
• ECS categorisation: event.kind, event.category, event.type and event.outcome hold "the 'What it is' information, independent of the source of the events"; leave them empty when no allowed value fits.
• ECS and OpenTelemetry converging: in April 2023 Elastic and OpenTelemetry announced a plan to merge ECS into OpenTelemetry semantic conventions as "a single open schema" maintained by OpenTelemetry.

Making normalisation reliable

1. Pick one target schema per destination and document the version; record it on every event.
2. Write mapping tests with sample events per source, including failures and multi-line records, run in CI on every change.
3. Route parse failures with the raw event to a dead-letter destination and alert on the rate; never drop them silently.
4. Check detections against the schema: a renamed field silently breaks every rule using it, so keep a field-usage inventory from the detection repository. See the Detection Engineering Guide.
5. Keep two clocks: store event time and ingest time in UTC, and flag events whose drift exceeds a threshold.

Enrichment

Enrichment at the pipeline adds context once, before storage, so every downstream tool sees it. The trade-off is staleness: context written into an event is a snapshot, while query-time lookups reflect the current state.

• Enrich at ingest: asset criticality from the CMDB or cloud tags, and account type (human, service account, workload identity, AI agent) and privileged groups from the IdP. These drive routing and turn "logon from new host" into "service account used interactively".
• Enrich at query time: threat intelligence matches, because indicators expire and should not be stamped permanently into evidence. See the Threat Intelligence Guide.
• Fail open, and say so: if a lookup source is down, pass events through tagged as unenriched, never drop them, and record lookup database versions.

Filtering, sampling and volume reduction

Volume reduction is the usual business case for a pipeline, and the usual way pipelines blind a SOC. Order matters too: OpenTelemetry notes that "the order of processors dictates the order in which data is processed".

• Every drop rule is a detection decision: give it an owner, a reason, a review date and a test showing detections are unaffected.
• Reduce the copy, not the evidence: route the full stream to cheap storage and the reduced stream to the SIEM, so an investigation can still reach the original.
• Redact deliberately: logs carry personal data and secrets pasted into command lines. The OpenTelemetry redaction processor removes attributes not on an allow list; mask secrets before they spread to every store. See the Log Management and Retention Guide for retention and privacy obligations.

Routing, tiering and the security data lake

Routing turns one SIEM feed into purpose-built streams. Decide per source and event class which destination it serves and what reduction is allowed, and express it as configuration.

A security data lake stores normalised security data in open, cheap formats for long-term search. Amazon Security Lake, for example, converts ingested data to Apache Parquet and OCSF, stores it in S3 buckets the customer owns, and grants subscribers either data access (objects delivered with notifications) or query access through Athena, Redshift and Glue-integrated tools, with one Lake Formation table per source. For any lake, check:

• Query path: can detections query the lake directly, or must data be rehydrated into the SIEM first, and how long does that take?
• Schema and access: a versioned schema so year-old hunts still run, and per-source, per-region access, because the lake holds the most sensitive telemetry you own.
• Cost: storage is cheap but scan-priced queries punish wide hunts; partition by time and source. A lower bill with fewer firing rules means evidence was cut.

Securing the pipeline itself

An attacker who controls the pipeline does not need to clear logs on every host. MITRE ATT&CK T1685 Disable or Modify Tools (tactic TA0112 Defense Impairment) includes "interfering with logging pipelines and forwarding mechanisms (e.g., SIEM ingestion)" and disrupting "logging infrastructure (e.g., syslog), or event aggregation". ATT&CK records that Salt Typhoon "has disabled logging and disabled the distribution of logs to log aggregators", one reason the Salt Typhoon telecom intrusions went unseen. Cloud variants are T1685.002 Disable or Modify Cloud Log.

• Encrypt and authenticate every hop: TLS with certificate validation on each hop from agent to destination.
• Minimise components: OpenTelemetry advises enabling only what you need, which "minimizes the attack surface exposed".
• Run unprivileged: "Avoid running the Collector as a root user."
• Keep secrets out of configuration files: collector configuration can contain "API tokens" and "TLS certificates including private keys"; reference them from a secrets manager or injected environment (${env:VAR}), never the repository.
• Change-control the configuration: configuration as code, peer review for any filter, drop or route change, CI deployment and a deployable previous version. A one-line drop rule is the quietest way to blind a SOC.
• Protect integrity downstream: write archive copies to immutable storage administered separately from the pipeline, and stamp collector ID and pipeline version on each event for chain of custody.
• Watch the watchers: ATT&CK mitigation M1054 suggests "automatically relaunching forwarding mechanisms at recurring intervals"; alert on collector stops, configuration changes and per-source volume drops.

Where identity fits: the pipeline's non-human identities

A pipeline is run almost entirely by non-human identities that can read all security telemetry and decide what the SOC sees. Inventory them as privileged access.

• Collector and agent service accounts: need read access to local logs and nothing else. Alert on interactive use. See the Service Account Security Guide.
• API clients that pull SaaS and cloud audit logs: OAuth clients and API keys with tenant-wide audit rights expose everyone's activity. Scope them to audit read, restrict source IPs where possible, and rotate them. See the API Key Management Guide.
• Ingest tokens: SIEM ingest tokens (Splunk HTTP Event Collector tokens are a common example) let the holder inject false events or flood a source. Issue one per collector group so each can be revoked without an outage.
• Cloud roles for lakes and buckets: prefer workload identity federation over static access keys for collectors running in cloud or Kubernetes, and give writers no delete rights on archive storage. See the Cloud Workload Identity Guide.
• Pipeline administrators: anyone who can change routing is a detection administrator; put them behind privileged access controls.

Implementing a pipeline step by step

1. Inventory: every source with daily volume, format, transport, destination, retention and dependent detections.
2. Define the target: schema per destination, raw-copy location, routing matrix and reduction rules allowed per data class.
3. Deploy in pass-through mode: dual-write untransformed events to the existing destination and the new path; compare per-source counts daily until they match.
4. Add parsing and normalisation: one source group at a time, with mapping tests in CI and parse failures routed to a dead-letter store.
5. Add reduction in shadow mode: send both full and reduced streams to a test index, run the detection set against both, and only enable a reduction when rule results match.
6. Cut over by wave: identity and EDR sources first, then cloud, then network. Keep the direct path configured but disabled.
7. Roll back by routing: re-enable the direct path or redeploy the previous version from source control; test this before the first cut-over.
8. Operate: health dashboards, discard and parse-failure alerts, quarterly review of every drop rule and every pipeline credential.

Decision guidance

• SIEM-native ingestion or a separate pipeline: if one SIEM receives everything at acceptable cost, its own connectors are simpler and carry fewer credentials. A separate pipeline earns its place with several destinations, a planned migration, or ingest cost to cut without losing raw data.
• Open source or commercial: OpenTelemetry Collector, Vector and Fluent Bit need engineers who run the pipeline as a product. Commercial pipelines add management planes and packaged parsers; check how they store your credentials and whether rules are exportable.
• Shared collectors: one agent for security and observability lowers overhead but couples two teams' changes; agree who owns the security routes.
• Small organisations: use built-in SIEM or XDR collectors and avoid reduction logic you cannot test.
• Large organisations: a platform team owning the pipeline, a durable queue, OCSF or ECS as the internal contract, a security data lake for raw and historical data, and detection engineers signing off every reduction rule.

Detection and measurement

Send pipeline health telemetry to a destination the pipeline does not control, and track:

• Source silence and volume drift per source, against a learned baseline.
• Discarded events from buffer overflow or failed delivery, reported separately from intentional drops.
• Parse failure rate and schema conformance (required fields populated) per source.
• End-to-end latency at the 95th percentile, inside the shortest correlation window.
• Cost per GB and reduction ratio by source, read alongside the detections each source supports.

Framework mapping

Common failure patterns

• Reduction approved by finance, not detection: "noisy" authentication failures or DNS are dropped, and spraying and beaconing rules stop firing without an error.
• Memory buffers on security sources: a SIEM outage loses hours of audit data, found only during an investigation.
• Schema change breaks rules silently: a parser update renames a field, and every rule referencing the old name matches nothing.
• Unauthenticated collectors on every interface: anyone on the network can inject events or flood the pipeline into discards.
• Pipeline credentials in plain-text config: an audit API key with tenant-wide read rights sits in a Git repository or container image.

Practitioner checklist

• Keep pipeline configuration in version control with peer review and CI tests for parsers, mappings and drop rules.
• Normalise to one documented schema per destination, keep the raw event and dead-letter parse failures.
• Use durable buffers for security sources and alert on any discard.
• Never sample authentication, audit, EDR or cloud control plane events.
• Test every reduction rule against the detection set in shadow mode before enabling it.
• Encrypt and authenticate every hop (syslog over TLS on 6514, OTLP with TLS), bind listeners to specific interfaces.
• Inventory, scope, vault and rotate collector accounts, audit API clients and ingest tokens.
• Alert on source silence, volume drift, collector stops and pipeline configuration changes.

Standards and references

• OCSF: Open Cybersecurity Schema Framework
• OCSF: Understanding OCSF
• OpenTelemetry: Semantic Conventions
• Elastic: Elastic Common Schema (ECS) Reference
• OpenTelemetry: Collector Configuration Best Practices
• IETF: RFC 5425 Transport Layer Security (TLS) Transport Mapping for Syslog
• NIST: SP 800-92 Rev. 1 Cybersecurity Log Management Planning Guide (initial public draft)
• MITRE ATT&CK: T1685 Disable or Modify Tools

Related NHI Mgmt Group resources: SIEM Guide · Log Management and Retention Guide · Detection Engineering Guide · Secrets Management Guide