Log Pipeline Migration Playbook: Collectors and Forwarders

Replacing the collectors, forwarders and routing layer that carry logs to your SIEM and data lake can silently drop exactly the events your detections and investigations depend on, and nothing in the destination tells you what never arrived. Organisations migrate log pipelines to cut ingest cost, add a vendor-neutral routing layer in front of several destinations, retire unsupported forwarders, move to an open protocol, or consolidate pipelines after a merger. The pipeline sits on the path of every security log, holds credentials for every source and every destination, and decides through its filters what the security team will ever see. This playbook covers what moves, the cutover patterns, delivery guarantees and buffering, parsing and field parity, filtering decisions, the identities involved and how to prove that nothing was lost. It is vendor-neutral and draws on the joint ASD's ACSC and partner agency guidance on event logging, OMB M-21-31, NIST SP 800-92 Rev. 1 (draft) and the OpenTelemetry specifications. Building a pipeline for the first time is covered in the Log pipeline (collectors, forwarders, data pipeline) rollout playbook, and hardening it in Securing the log pipeline itself.

Key takeaways

Prove completeness, do not assume it. Count events per source at the edge and at every destination during the overlap; an unexplained difference is a detection gap.
Field parity matters more than event parity. An event that arrives with a renamed or unparsed field breaks the rules that read it just as surely as a dropped event.
Buffer to disk at the edge. In-memory queues lose data on restart or during destination outages; persistent queues and replay capability are part of the design, not an option.
Filtering is a security decision. Every drop, sample or reroute rule needs an owner and a record, because it decides what investigators will never be able to see.
The pipeline holds keys to everything. Source read credentials, destination write tokens and TLS keys are non-human identities: reissue, scope and revoke them as part of the move.

What a log pipeline migration actually moves

Component	Migration approach	Watch for
Endpoint and server agents (forwarders)	Replace by ring; run old and new side by side on pilot hosts	Agents reading the same file twice, or neither reading it after a handover
Syslog and network device collection	Repoint devices or put the new collector behind the existing address	UDP syslog with no delivery confirmation; devices that cannot be repointed quickly
API pullers for SaaS and cloud audit logs	New connector with its own credentials; overlap the polling windows	Gaps or duplicates at the switch, because checkpoints do not transfer
Aggregation and routing tier	Build in parallel; route to all destinations from both tiers during overlap	Routing rules that sent a source to the archive but not to the SIEM
Parsers, normalisation and enrichment	Re-implement per source; test against captured samples	Timestamp and time zone handling, multi-line events, field renames
Filters, sampling and drop rules	Review each with an owner; migrate only the justified ones	Undocumented drops that removed security events to save cost
Buffers and queues	Drain old queues before shutdown; size new persistent queues	Events stranded on disk when an old forwarder is uninstalled
Credentials and certificates	New scoped credentials per source and destination	Shared tokens copied into the new configuration
Pipeline health telemetry	Rebuild health alerts before cutover	No alert when a collector stops or a queue fills

The joint event logging guidance published in 2024 by ASD's ACSC with CISA, the FBI, NSA, NCSC-UK and other partners recommends prioritising logs from critical systems, internet-facing services, identity and domain servers, edge devices, administrative workstations, privileged systems and cloud tenants. Use that list to rank sources for the migration: these move last, with the longest overlap and the strictest parity tests.

Cutover patterns

Pattern	How it works	Choose it when	Trade-off
Dual collection	Old and new agents both collect and send; compare at the destination	Critical sources where parity must be measured	Double ingest cost and host load during overlap; duplicates must be tagged
Fan-out from the new tier	New collectors send to old and new destinations	Destinations are also changing, or several destinations share data	Old destination depends on the new pipeline, so it no longer gives an independent comparison
Source-by-source cutover	Switch one source type at a time with a short overlap	Large estates with many source types and limited budget	Longer programme; mixed estate for months
Transparent replacement	New collector takes over the old address or DNS name	Thousands of network devices that cannot be reconfigured	Rollback must restore the old service at the same address

Tag every event with the pipeline that carried it (for example a pipeline field set to old or new) during overlap, so detections can deduplicate and analysts can compare.

Delivery guarantees, buffering and back-pressure

A pipeline loses data in three places: at the source when it cannot keep up, in memory when a component restarts, and at the destination when it rejects or throttles data. Design for each.

Persistent queues: the OpenTelemetry Collector's exporter helper documents that data which cannot be added to the sending queue is typically dropped, and that a persistent queue backed by a storage extension lets the collector resume exporting after it is killed and restarted. The same principle applies to any collector: buffer to local disk, size the buffer for your longest expected destination outage, and alert when it passes 70% full (NHI Mgmt Group recommendation).
Honour back-pressure: the OTLP specification defines HTTP 429, 502, 503 and 504 as retryable, asks clients to honour a Retry-After header and to back off exponentially, and allows servers to report partial success. A new pipeline that ignores these signals either floods a recovering destination or discards data.
Reliable transport: replace plain UDP syslog with TCP and TLS where devices support it. OMB M-21-31 requires US federal agencies to encrypt logs in transit and to be able to replay original logs.
Drain before decommission: stop inputs on old forwarders, wait for queues to empty, confirm in the destination, then uninstall.

Parsing, timestamps and field parity

Detections and dashboards read fields, not raw lines. Build a parity test per source type from captured samples of real events, including rare event types.

Capture a sample set: several days of raw events per source type, including errors, multi-line events and daylight saving transitions.
Run both parsers: process the same samples through old and new parsing and diff the resulting fields: names, types, values, and the event timestamp.
Fix timestamps first: the joint ASD's ACSC guidance recommends UTC and ISO 8601 with millisecond precision across systems; M-21-31 sets a similar requirement for federal agencies. Keep the original event time and the ingest time as separate fields, so delayed delivery is visible.
Map to a schema once: if you normalise to an open schema such as OCSF, do it in the pipeline so every destination gets the same fields. See the OCSF Guide.
Re-run detections: run the SIEM rules that depend on each source against data from both pipelines and compare results.

Time quality is a pipeline dependency too: if collectors re-stamp events with their own clock, a drift there corrupts every timeline. See the NTP Security Guide.

Filtering, routing and retention decisions

A migration is often justified by cost savings from filtering. The joint guidance supports filtering logs before they reach the SIEM, while sending full data to a central store such as a data lake. Make that split explicit:

Decision	Safe pattern	Risky pattern
Reduce SIEM volume	Route low-value events to a cheaper searchable tier or archive	Drop them at the edge with no copy anywhere
Sampling	Sample high-volume performance data only	Sample authentication, process creation or audit events
Field trimming	Remove verbose fields that no rule or investigation uses, after review	Remove command lines or source addresses to save space
Sensitive data	Mask tokens and personal data in the pipeline with an approved rule set	Forward secrets unchanged to every destination
Retention	Set by a written policy and risk assessment per log type	Inherit whatever the destination plan defaults to

For reference points on retention, M-21-31 sets minimums for US federal agencies of 12 months in active storage and 18 months in cold storage for most log categories, and CIS Controls v8.1 Safeguard 8.10 sets a minimum of 90 days. The joint guidance notes that incidents can take up to 18 months to discover. See the Log Management and Retention Guide.

Where identity fits

Source credentials: API pullers for SaaS and cloud audit logs use OAuth clients, service principals or API keys, often with broad read rights. Create new credentials for the new pipeline with read access to audit logs only, and revoke the old ones when the overlap ends. See the API Key Management Guide.
Destination tokens: use a separate write token per pipeline tier and destination, so a compromised edge collector cannot write to, or read from, everything.
Collector privileges: agents that read protected logs need some privilege, but not full administrative rights. The OpenTelemetry Collector security guidance says the collector should not run as root or admin and that privileged requirements must be documented.
Secrets in configuration: keep credentials and TLS private keys in a secrets store, not in configuration files distributed to every host.
Mutual TLS between tiers: authenticate collectors to aggregators with certificates issued per host or per tier, so that a rogue sender cannot inject events.

See the Service Account Security Guide and the Non-Human Identity (NHI) Guide.

The phased playbook

Inventory: every source, its collection method, volume per day, destinations, parsers, filters and credentials. Mark sources that feed detections.
Baseline: record events per source per hour at the destination for at least two weeks (NHI Mgmt Group recommendation), so you know what normal looks like.
Design: tiers, persistent buffering, transport encryption, credential layout, schema, filter and routing rules, retention, and health telemetry.
Pilot: one low-risk source type, end to end, with parity tests, a forced destination outage to test buffering, and a rehearsed rollback.
Migrate in rings: low-value sources first, high-value sources from the priority list last, each with dual collection and a parity sign-off.
Cut over: stop old collection for a source only after its counts, fields and detection results match within tolerance for at least one full week.
Drain and decommission: drain queues, uninstall old agents, close old listener ports, and revoke old credentials and certificates.
Review: confirm filters and retention against policy, and record the final source inventory.

Rollback: during dual collection, rollback is stopping the new agent. For transparent replacements, keep the old collector's configuration and image ready to restore at the same address. Never uninstall the old forwarder from a host until its new agent has passed parity, and keep old destination tokens valid (but monitored) until the source is signed off.

Small vs large organisations: a small estate with a single SIEM and a few source types can often run dual collection for everything for two weeks. Large estates need an aggregation tier, ring rollout, automated per-source count comparison and a filter review board.

Detection and measurement

Metric or signal	Why it matters	Target (NHI Mgmt Group recommendation)
Event count difference per source, old vs new pipeline	Measures loss or duplication	Below 0.5% for priority sources, every gap explained
Field parity per source type	Detections read fields	100% of fields used by rules populated
Ingest lag (event time to index time)	Late events break correlation and alert timing	Within the SLA set for each source
Sources silent beyond their expected interval	A stopped collector looks like a quiet system	Alert within one interval; zero unexplained
Persistent queue fill and dropped event counters	Early warning of loss	No drops; alert at 70% full
Old credentials and listener ports still active after cutover	Forgotten paths into the pipeline	Zero within 30 days

M-21-31 asks agencies to monitor for logging disruptions and send near real-time alerts when logging stops unexpectedly. Disabling event logging and tampering with security tools are tracked in MITRE ATT&CK as T1685 Disable or Modify Tools and its sub-technique T1685.001 Disable or Modify Windows Event Log; a migration window, with agents being removed and reinstalled, is the easiest time for such activity to hide. The Storm-0558 key breach is a reminder that whether a particular audit event is collected at all can decide whether an intrusion is ever noticed.

Common failure patterns

Checkpoints not carried over: an API puller starts "from now" on the new pipeline, leaving a gap of hours or days of SaaS audit logs.
Timestamp drift: the new parser uses ingest time or the wrong time zone, so events appear hours away from related activity.
Silent cost filters: a drop rule added to hit a budget removes security events, and nobody records it.
Stranded buffers: old forwarders uninstalled with unsent events still queued on disk.
Shared tokens: one destination write token copied to every collector, impossible to rotate without an outage.
No health alerts on the new tier: the first sign of a dead collector is a missing log during an incident.

Framework mapping

Activity	NIST SP 800-53 Rev 5	NIST CSF 2.0	CIS Controls v8.1
Log generation and source coverage	AU-2, AU-12	PR.PS-04	8.2
Buffer and storage capacity	AU-4	PR.PS-04	8.3
Alerting on logging failures	AU-5	DE.CM-09	8.2
Time stamps	AU-8	PR.PS-04	8.4
Protection of logs in the pipeline	AU-9, SC-8	PR.DS-02	8.9
Retention	AU-11	PR.DS-01	8.10
Change control for filters and routing	CM-3	PR.PS-01	Not mapped

Practitioner checklist

Inventory every source, collection method, parser, filter, destination and credential.
Baseline per-source event counts before any change.
Use persistent, disk-backed buffers and test them with a forced destination outage.
Diff parser output on captured samples, timestamps first.
Carry over or overlap API pull checkpoints to avoid gaps.
Give every filter, sample and drop rule an owner and a record.
Issue new scoped source and destination credentials; use mutual TLS between tiers.
Build pipeline health alerts before the first production cutover.
Drain old queues before uninstalling forwarders.
Revoke old credentials and close old listeners on a dated plan.

Log pipeline (collectors, forwarders, data pipeline) migration playbook: moving platforms without losing control