Log pipeline (collectors, forwarders, data pipeline) rollout playbook: phased implementation with security built in

The log pipeline sits between every log source and every detection, so a pipeline that drops, delays, alters or leaks events quietly undermines the SIEM, the data lake and any investigation that depends on them. This playbook covers building a log pipeline as a product class, meaning the collectors and agents on sources, the forwarders and aggregation tier, and any stream or data pipeline layer that filters, normalises and routes events. It sets out the design decisions, the order in which to onboard sources, transport and identity controls, delivery guarantees, normalisation and time, filtering and retention, go-live with rollback, and the measures that prove nothing is being lost. It is vendor-neutral and based on the multi-agency Best Practices for Event Logging and Threat Detection, OMB M-21-31, RFC 5425, OpenTelemetry Collector documentation and NIST SP 800-53. It is written for detection engineers, SIEM and platform engineers and security architects. Hardening the pipeline is covered in Securing the log pipeline (collectors, forwarders, data pipeline) itself, and moving between pipelines in the Log pipeline (collectors, forwarders, data pipeline) migration playbook.

Key takeaways

• Onboard sources by risk, not by ease. The joint guidance puts critical systems, internet-facing services, identity servers and edge devices ahead of the long tail of easy sources.
• Encrypt and authenticate every hop. The joint guidance asks for TLS 1.3 transport; for syslog, RFC 5425 defines TLS on TCP 6514 with certificate authentication of both ends.
• Design for loss before it happens. Buffers fill and queues drop data by default when full; persistent queues, capacity headroom and alerts when a source goes quiet are rollout requirements.
• Normalise time and schema at the pipeline. UTC, ISO 8601 with milliseconds and a consistent structured schema make correlation possible across sources.
• Filter deliberately and keep the raw copy. Reduce what reaches the SIEM for cost, but keep a full-fidelity copy in cheaper storage for the retention period your risk assessment sets.

Phase 0: architecture decisions

Decide retention before sizing. The joint guidance recommends basing retention on risk assessment and notes incidents can take up to 18 months to discover; OMB M-21-31, binding on US federal agencies, sets minimums of 12 months in active storage and 18 months in cold storage for most log categories. See the Log Management and Retention Guide and the Security Data Lake Guide.

Phase 1: source inventory and onboarding order

The Best Practices for Event Logging and Threat Detection, published by the Australian Signals Directorate's ACSC with CISA, FBI, NSA, NCSC-UK and other partners, lists priority enterprise log sources. Use it as the onboarding order:

For each source, record owner, collection method, expected events per second, required fields and the detections that depend on it. The SIEM Log Source Onboarding Checklist gives a per-source template.

Phase 2: transport, identity and access

• Encryption in transit: the joint guidance recommends TLS 1.3 for log transport, and OMB M-21-31 requires log data to be encrypted between source and destination. For syslog sources, use RFC 5425 TLS transport (default TCP port 6514), which requires both sender and receiver to implement certificate-based authentication and recommends mutual authentication as the default policy. Retire plain UDP syslog wherever the source supports TLS.
• Collector and agent credentials: each agent, forwarder and API pull job is a non-human identity. Use per-tier certificates or tokens, issued from your PKI or secret store, scoped write-only to the next hop, with an owner and rotation date. API pulls from SaaS and cloud services need read-only scopes for audit logs, not administrator roles.
• Least privilege on hosts: OpenTelemetry's Collector guidance is to avoid running as root and document any component that needs elevated rights; the same applies to any agent that only needs to read logs.
• Bind and segment: receivers listen only on the interfaces they need, and forwarders sit on a segmented logging network. The joint guidance recommends storing logs on a separate or segmented network.
• Configuration control: pipeline routing and filter configuration lives in version control with review; a filter that drops a field is a detection change (NIST SP 800-53 CM-3).

The Sumo Logic breach is the reason to inventory pipeline credentials from day one: after a compromised credential was used to access one of the log platform's AWS accounts, customers were advised to rotate API access keys and collector credentials. See the Secrets Access Logging and Detection guide for monitoring those credentials.

Phase 3: delivery guarantees and capacity

Most pipeline components drop data under pressure unless configured otherwise. In the OpenTelemetry Collector, for example, data that cannot be added to a full sending queue is typically dropped, the default queue holds 1,000 batches, and a persistent queue backed by a storage extension lets a restarted collector resume exporting what it had buffered. Equivalent settings exist in other agents and brokers.

1. Size for peaks: measure events per second per source in Wave 1 and plan capacity for incident-time spikes, when logging volume and investigative value are highest.
2. Use disk-backed buffering at forwarders and for high-value sources so a destination outage of hours does not lose data.
3. Prefer acknowledged delivery: choose protocols and settings where the receiver acknowledges receipt and the sender retries with back-off.
4. Expose pipeline metrics: queue depth, enqueue failures, dropped and refused records and export latency, scraped and alerted like any production service.
5. Test failure on purpose: stop a destination during the pilot and confirm data is buffered and replayed, and that alerts fire.

Phase 4: normalisation, time and enrichment

• Time: the joint guidance asks for a single accurate time source, UTC and ISO 8601 with milliseconds (for example 2024-07-25T20:54:59.649Z). Keep the original source timestamp and add a pipeline receipt timestamp so delays are measurable. See the NTP Security Guide.
• Schema: normalise to a consistent structured format such as JSON with a common schema. The OCSF Guide covers one open schema option.
• Required fields: OMB M-21-31 lists minimum fields where applicable, including timestamp, event status code, device identifier, session or transaction ID, source and destination IP, user ID, command executed and a unique event identifier. Check parsers populate them before declaring a source onboarded.
• Sensitive data: mask secrets and tokens that leak into logs at the earliest tier, and restrict access to logs that contain sensitive data, as the joint guidance advises. See the Privacy in Security Logging and Monitoring Guide.

Phase 5: filtering, routing and retention

The joint guidance recommends filtering before sending logs to a SIEM or XDR to manage cost and capacity, while keeping logs that confirm intrusion and impact for longest. OMB M-21-31 asks for integrity checks of closed log files against stored hashes for the retention period. Every filter rule should name the detections it affects; a rule with no named impact is not reviewed properly.

Phase 6: go-live, rollback and handover

• Parity check per source: before switching a source's detections to the new pipeline, compare event counts over the same window at source, pipeline and destination; investigate any difference above an agreed tolerance (NHI Mgmt Group recommendation: under 0.1 percent for Wave 1 sources).
• Rollback: keep the previous path, or direct forwarding, deployable per source until the new route has run through a month-end or other peak; roll back source by source, not all at once.
• Logging failure alerts: OMB M-21-31 requires near real-time alerts to responsible parties when logging stops unexpectedly. Build source silence alerts per source with thresholds based on its normal volume.
• Handover: source owners, parser ownership and capacity reviews move into day-2 operations, with a periodic Log pipeline (collectors, forwarders, data pipeline) health check.

Decision guidance

• Agent or agentless: agents suit servers and endpoints where local buffering and filtering matter; agentless API collection suits SaaS and cloud services where no agent is possible. Use one standard agent where you can, since every agent type is another privileged binary to patch.
• Broker or direct forwarding: add a message broker when several destinations consume the same data or destination outages are common; otherwise disk-backed forwarders are simpler. A broker brings its own access control and retention to secure; see Securing the message broker and event streaming platform itself.
• Small organisations can rely on a SIEM vendor's or cloud provider's collectors with TLS and a single forwarder, focusing on Wave 1 sources and silence alerts. Large organisations need a regional forwarder tier, a data lake for full-fidelity retention and a pipeline team that owns parsers and capacity.

Detection and measurement

Attackers who reach a pipeline can stop or reconfigure collection, mapping to ATT&CK T1685 Disable or Modify Tools and, on Windows, T1685.001 Disable or Modify Windows Event Log, or tamper with events in transit, which falls under T1565 Data Manipulation. Monitor configuration changes to collectors and routes, new or changed pipeline credentials, agents stopped on hosts and sudden drops in source volume.

Common failure patterns

• Easy sources first: the SIEM fills with firewall and proxy volume while identity servers and remote access gateways are still not onboarded.
• Plain UDP syslog left in place: unauthenticated, unencrypted and lossy, so spoofed or missing events go unnoticed.
• Default in-memory queues: a destination outage or peak drops events with only a metric nobody watches.
• Filters without impact analysis: a cost-saving rule drops the field a detection depends on.
• Local time and mixed formats: correlation across sources fails during the incident that needed it.
• One shared pipeline credential: the same token on every collector, impossible to rotate quickly after a vendor or host compromise.

Framework mapping

ISO/IEC 27001:2022 Annex A control A.8.15 Logging covers producing, storing and protecting logs, with A.8.17 Clock synchronisation for time sources. Federal agencies should also map the rollout to the event logging tiers EL1 to EL3 in the OMB M-21-31 Logging Maturity Guide.

Practitioner checklist

• Set retention by risk assessment and regulatory need before sizing storage.
• Choose collection, aggregation, buffering and destination layers and name a pipeline owner.
• Onboard sources in the joint guidance priority order with an owner per source.
• Use TLS 1.3 where supported and RFC 5425 TLS syslog with certificate authentication; retire UDP syslog.
• Issue per-tier pipeline credentials with owners, scopes and rotation dates.
• Enable disk-backed buffering and alert on queue drops and enqueue failures.
• Normalise to UTC ISO 8601 with milliseconds and a consistent schema with required fields.
• Route filtered events to the SIEM and a full-fidelity copy to the data lake, with integrity hashes.
• Run per-source parity checks and keep per-source rollback until after a peak period.
• Alert within minutes when any source goes quiet.

Standards and references

• ASD's ACSC, CISA, FBI, NSA, NCSC-UK and partners: Best Practices for Event Logging and Threat Detection
• OMB: M-21-31 Improving the Federal Government's Investigative and Remediation Capabilities Related to Cybersecurity Incidents
• IETF: RFC 5425 Transport Layer Security (TLS) Transport Mapping for Syslog
• OpenTelemetry: Collector exporter helper, sending and persistent queues
• OpenTelemetry: Collector configuration best practices
• OWASP: Logging Cheat Sheet
• NIST: SP 800-53 Rev. 5 Security and Privacy Controls
• CIS: Critical Security Controls v8.1

Related NHI Mgmt Group resources: SIEM rollout playbook · CIS Control 8 Audit Log Management Guide · If your log pipeline (collectors, forwarders, data pipeline) is compromised · Monitoring and observability platform rollout playbook