When the log pipeline is compromised, the attacker controls what your SOC sees: events can be dropped, altered or invented before they reach detection, and the pipeline's own credentials open the doors to every source and destination it touches. Collectors, forwarders and stream processing tiers sit between every log source and the SIEM, data lake or archive. They hold destination tokens, TLS private keys and cloud roles that read audit logs across accounts, and they often run with elevated rights on the hosts they collect from. This guide is a response playbook for a suspected or confirmed compromise of that pipeline. It covers detection, triage, containment, credential rotation, integrity recovery, rebuild and evidence, for detection engineers, platform teams and incident responders. It is vendor-neutral and applies to agent-based, agentless and managed pipelines.
Key takeaways
- The first casualty is trust in your telemetry. Until you prove otherwise, every detection that depended on the pipeline during the exposure window may be incomplete or false.
- Recover integrity from the edges. Source-side logs and native cloud audit trails that bypass the pipeline are how you rebuild an accurate timeline and backfill gaps.
- Pipeline credentials are high-value non-human identities. Ingest tokens, TLS client certificates, cloud reader roles and queue credentials must be rotated in a planned order.
- Configuration is the attack surface. Filter, routing and transform rules are where data is silently dropped or diverted, so diff them against version control first.
- Rebuild from code, not from the running estate. Redeploy collectors and processors from reviewed configuration and verified packages instead of cleaning them in place.
What the pipeline holds and why attackers target it
A pipeline compromise serves two aims. The first is defence impairment: removing or altering the events that would reveal the intrusion, which ATT&CK groups under the Defense Impairment tactic (TA0112) and techniques such as T1685 Disable or Modify Tools. The second is access: the pipeline's credentials and the sensitive content flowing through it. Altering stored logs to mislead investigators also fits T1565.001 Stored Data Manipulation.
| Tier | Credential or privilege held | Attacker use |
|---|---|---|
| Host collectors and agents | Root or SYSTEM, read access to all local logs, destination token or client certificate | Stop or filter collection on a host, harvest secrets in local logs, impersonate the host to the destination |
| Aggregation and forwarding tier | Listener TLS keys, destination tokens for SIEM, lake and archive | Drop or rewrite events in transit, send copies to an attacker endpoint, inject false events |
| Cloud log readers | Cross-account roles or keys reading audit logs, storage buckets and queues | Read audit trails and data across accounts (T1078.004 Cloud Accounts) |
| Message queue or streaming layer | Producer and consumer credentials, topic administration rights | Delete topics, change retention, consume all events |
| Pipeline management plane | Admin users, API keys, fleet configuration push | Change routing for the whole fleet, push new components or scripts (T1072 Software Deployment Tools) |
| Configuration repository | Write access, deploy keys | Plant a malicious filter that survives redeployment |
The OpenTelemetry collector security guidance makes the general point for this tool class: collector configuration can contain API tokens and TLS certificates with private keys, which belong in an encrypted file system or a secret store rather than plain configuration files. See the Securing the log pipeline (collectors, forwarders, data pipeline) itself guide for the preventive baseline.
How the compromise shows up
| Signal | How to spot it | Likely meaning |
|---|---|---|
| Volume drop for one source type, host group or event ID | Per-source event count baselines in the SIEM; compare with source-side counts | Filter rule added or collector stopped |
| New destination, output or route in pipeline configuration | Configuration diff against version control; outbound connections from pipeline hosts | Exfiltration of log data or duplicate stream to attacker |
| Events with impossible timestamps, wrong host metadata or missing sequence numbers | Parsing error dashboards, time skew checks | Injected or rewritten events |
| Changes to retention, buffering or queue topics | Queue and storage audit logs | Evidence destruction |
| Cloud log reader role making write or unusual read calls | Native cloud audit logs for the reader principal | Stolen pipeline credential |
| Log clearing on Windows sources | Security event 1102 and System event 104 at source | Source-side tampering (T1685.005 Clear Windows Event Logs) |
Self-hosted log analytics and search tiers also appear in the CISA Known Exploited Vulnerabilities Catalog, for example CVE-2026-20253 (missing authentication for a critical function, added June 2026) and older remote code execution entries such as CVE-2015-1427 and CVE-2019-7609. An exposed management interface on an affected version should trigger this playbook.
Triage questions
- Which tier is affected? Host collectors, the aggregation tier, cloud readers, the queue, the management plane or the configuration repository.
- When did the telemetry stop being trustworthy? Find the earliest configuration change or volume anomaly. That date defines the exposure window for every downstream detection.
- Which detections relied on the affected path? List the detection rules that consume the affected sources so the SOC knows which alerts could have been suppressed.
- Is there an independent copy? Identify source-side retention (host logs, device buffers, cloud audit trails, immutable archive) that bypassed the compromised tier.
- Did data leave? Check for new outputs and outbound traffic from pipeline hosts, and whether the affected streams carry credentials or personal data.
- Who changed what? Pull management plane and repository audit history and match each change to a ticket.
Containment by priority
- Snapshot the evidence: export running configuration from every pipeline tier, management plane audit logs, repository history and queue metadata. Image aggregation servers before rebuilding them.
- Freeze configuration change: lock the configuration repository to the response group, disable fleet configuration push and pause automated deployments of pipeline components.
- Cut unauthorised outputs: block egress from pipeline hosts to any destination not on the approved list, and remove unknown outputs and routes.
- Open a bypass path: send critical sources (identity provider, cloud audit logs, EDR, domain controllers) directly to the SIEM or an immutable store through a temporary, separately credentialed path so detection continues.
- Protect retention: extend retention on queues, buffers and source systems, and enable object lock or write-once storage on the archive so the attacker cannot shorten it.
- Revoke management access: remove unexpected admin users and API keys and require phishing-resistant MFA for the remaining response group.
Decision: keep the pipeline flowing or stop it. Stopping protects confidentiality but blinds detection. In most cases keep it flowing through clean, freshly credentialed components while the compromised tier is isolated, and accept a short duplicate-event period during cut-over.
Small and large organisations: a small team with one forwarding tier and a managed SIEM can usually rebuild the whole pipeline in a day and should do so rather than investigate each collector in place. A large estate with thousands of collectors, several regions and a streaming layer should contain by tier: isolate the aggregation tier first, protect the bypass for identity and cloud audit sources, then rotate collector credentials ring by ring through its deployment tooling. In both cases, decide up front who can approve a temporary loss of a log source, because containment will cause some gaps.
Credential rotation sequence
| Order | Credential | Notes |
|---|---|---|
| 1 | Management plane admins, API keys, repository deploy keys | Stops the attacker changing configuration again during rotation |
| 2 | Destination ingest tokens and HTTP event collector tokens | Issue one token per source class, revoke old tokens, watch for rejected writes |
| 3 | TLS server and client certificates on listeners and forwarders | If the issuing CA private key was on a compromised host, replace the CA and re-enrol every collector |
| 4 | Cloud log reader roles and keys | Replace static keys with role assumption, reduce read scope to audit and log resources only |
| 5 | Queue and streaming credentials | Separate producer and consumer identities; remove topic administration from runtime identities |
| 6 | Secrets seen in the log stream | Any token, password or key that passed through the compromised tier is exposed and must be rotated at its owner |
The Sumo Logic breach shows how a single compromised cloud credential at a log analytics provider led to customers being asked to rotate their own access keys. See the Credential Rotation at Scale guide.
Restoring telemetry integrity
- Gap analysis: for each source, compare event counts per hour at source with counts at the SIEM across the exposure window. Gaps tell you what was dropped.
- Backfill from source: re-ingest missing events from host logs, device buffers and native cloud audit trails into a separate, clearly labelled index so original and recovered data are not mixed.
- Re-run detections: replay the affected detection rules over the backfilled data and triage any new hits as part of this incident.
- Flag suspect data: mark events that passed through the compromised tier during the window as untrusted in case notes; investigators should corroborate them from a second source.
- Prove integrity going forward: use authenticated transport, per-source credentials and write-once archive storage. NIST SP 800-53 AU-9 (Protection of Audit Information), AU-9(2) (store on separate physical systems) and AU-9(3) (cryptographic protection) describe the target state.
Eradication and rebuild
- Review every filter, route, transform and output in version control and remove anything without an approved change.
- Rebuild aggregation and processing hosts from clean images and verified packages, then deploy reviewed configuration.
- Reinstall host collectors from verified packages through the endpoint or configuration management tool, with new per-source credentials.
- Recreate queues or topics where retention or access control was altered, and re-apply least-privilege access.
- Run in parallel with the bypass path until event counts match source counts for at least one full business cycle, then retire the bypass.
- Roll back by switching sources back to the bypass path if counts diverge; keep the bypass credentials valid until sign-off.
Evidence to preserve and notification
Preserve running configuration, repository history, management plane audit logs, queue metadata, host images of aggregation servers, egress flow records from pipeline hosts and the gap analysis itself. The gap analysis is often the evidence regulators and auditors ask for, because it shows what the organisation could and could not see. If the stream carried personal data or credentials to an attacker-controlled output, treat it as a data breach and involve privacy counsel early. Note that a pipeline compromise can also undermine evidence for other incidents, so tell the owners of any open investigation whose timeline relied on the affected sources.
Where identity fits
Every component in a pipeline authenticates as a non-human identity, and the sources most worth protecting are identity sources: identity provider sign-in and admin logs, directory security events and cloud audit logs. Give these sources their own credentials and a protected path so a compromise of the general pipeline cannot blind identity threat detection. See the Identity Telemetry Requirements guide and the Service Account Security Guide.
Detection and measurement
- Source health coverage: percentage of critical sources with a silence alert that fires within 30 minutes of no data.
- Count reconciliation: daily difference between source and destination event counts for critical sources, target under one percent.
- Configuration drift: number of running pipeline configurations that do not match version control, target zero.
- Credential granularity: percentage of sources using their own ingest credential rather than a shared token.
- Immutable retention: percentage of critical log volume landing in write-once storage.
Framework mapping
| Activity | NIST CSF 2.0 | NIST SP 800-53 Rev. 5 | MITRE ATT&CK |
|---|---|---|---|
| Detect volume and configuration anomalies | DE.CM-09, PR.PS-04 | SI-4, AU-6 | TA0112, T1685 |
| Contain and freeze configuration | RS.MI-01 | IR-4, CM-3, CM-5 | T1072 |
| Rotate pipeline credentials | RS.MI-02 | IA-5, SC-12 | T1552.001, T1078.004 |
| Recover integrity and backfill | RS.AN-03, RC.RP-03 | AU-9, AU-9(2), AU-9(3), AU-10 | T1565.001, T1685.005 |
| Plan and report | RC.RP-01 | IR-6, IR-8 | - |
In ISO/IEC 27001:2022 Annex A the closest controls are 8.15 (logging), 8.16 (monitoring activities) and 5.26 (response to information security incidents). In CIS Controls v8.1 the work maps to Control 8 (Audit Log Management).
Common failure patterns
- Trusting a quiet SIEM: no alerts during the exposure window is not evidence of no activity when the pipeline was filtering.
- One shared ingest token for everything: rotation becomes an all-or-nothing outage, so teams delay it.
- Cleaning configuration in place: a removed filter comes back with the next deployment because the malicious change was also in the repository.
- Self-signed CA on a collector host: if the CA key is on a compromised host, re-issuing leaf certificates achieves nothing.
- Forgetting secrets in the stream: tokens that passed through the pipeline in debug logs stay valid after the rebuild.
- No bypass path: teams switch the pipeline off and lose identity and cloud audit visibility for days.
Practitioner checklist
- Export running configuration from every tier and lock the configuration repository.
- Block egress from pipeline hosts to unapproved destinations and remove unknown outputs.
- Stand up a separately credentialed bypass for identity, cloud audit and EDR sources.
- Extend retention at sources, queues and archive, and enable write-once storage.
- Rotate management, ingest, TLS, cloud reader and queue credentials in order, replacing the CA if needed.
- Run per-source gap analysis, backfill from source and replay detections.
- Rebuild aggregation tiers from clean images and reviewed configuration.
- Add silence alerts and daily count reconciliation for critical sources.
Standards and references
- NIST: SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management
- NIST: SP 800-92 Rev. 1 (Initial Public Draft), Cybersecurity Log Management Planning Guide
- CISA and ASD ACSC: Best Practices for Event Logging and Threat Detection
- NIST: SP 800-53 Rev. 5
- NIST: Cybersecurity Framework 2.0
- OpenTelemetry: Collector configuration best practices
- MITRE ATT&CK: Disable or Modify Tools
- CISA: Known Exploited Vulnerabilities Catalog
Related NHI Mgmt Group resources: Securing the log pipeline (collectors, forwarders, data pipeline) itself · If your SIEM is compromised · If your monitoring and observability platform is compromised · Anti-Forensics Detection Guide