What happens when an OpenTelemetry Collector crashes while telemetry data is still buffered locally?

How crash recovery behaves when the collector had local persistence

When local persistence is enabled, the collector is designed to treat buffered telemetry as recoverable state rather than disposable in-memory data. After a crash, restart resumes from the last saved offset, so processing continues from the preserved checkpoint instead of replaying the whole buffer. That reduces duplication risk and makes short outages much less disruptive to downstream pipelines.

The practical implication is that durability depends on how the collector was configured, not on buffering alone. A memory-only buffer is lost on process death, while a persistent queue or on-disk state lets the collector pick up where it stopped. If the crash happened after some telemetry had already been flushed downstream, that data is not reprocessed unless the saved offset was behind the last successful send.

For operators, the key question is whether the restart path can trust the local state it finds. A clean resume assumes the disk state is intact and the collector can read the same queue metadata it wrote before the crash. If the storage layer was truncated, relocated, or mounted inconsistently, recovery may degrade into partial replay or loss even though persistence was enabled.

What telemetry delivery guarantees you can and cannot assume

Persistent buffering improves crash tolerance, but it does not turn the collector into an exactly-once system. At best, it gives you bounded replay from a saved checkpoint and a much lower chance of dropping in-flight telemetry during a restart. The remaining behaviour still depends on exporter semantics, batch handling, timeout behaviour, and whether the upstream producer keeps sending while the collector is unavailable.

That means the safe assumption is usually at-least-once delivery with possible duplicates, not perfect deduplication. If the downstream platform is sensitive to repeated spans or log records, you still need correlation IDs, idempotent ingestion where possible, and retention rules that tolerate retries. Buffered state protects continuity, but it does not remove all duplication paths after a failure.

This is why telemetry pipelines are often treated as operationally resilient rather than transactionally strict. In practice, the saved offset is the mechanism that keeps the collector from starting over, while the rest of the pipeline must be prepared for a brief interruption and a catch-up burst after recovery. That burst can matter if exporters, gateways, or backends have tight rate limits.

What to check after a collector crash

After a crash, the most useful verification is whether the collector restarted with the same persistent volume or data directory and whether it resumed at the expected checkpoint. If the queue depth drops to zero too quickly, or the backlog jumps unexpectedly, that can indicate partial replay, loss of state, or a downstream exporter that is throttling the catch-up flow. The recovery story is strongest when the restart is boring.

It is also worth checking whether the telemetry stream shows gaps that line up with the downtime window. A clean resume should leave a short interruption window, not a long hole in traces, metrics, or logs. If the collector sits behind a durable queue, compare the oldest buffered record against the crash time and confirm that the data retained on disk matches the outage period you expected to absorb.

For broader pipeline design, a guide to observability, telemetry and SRE for security teams is useful when you want to understand how buffering, sampling, and alerting fit together in a resilient collection path.

Practitioner Guidance

What to verify: Confirm that persistence is actually enabled, that the collector writes to a durable path, and that the restart reuses the same state directory or volume. A crash test is only meaningful if the storage survives the process restart.

Decision rule: If buffered telemetry can survive the crash on disk, treat the main risk as replay and backlog management; if it cannot, treat the crash as a data-loss event and shorten your recovery window accordingly.

What good looks like: The collector comes back with the preserved queue, continues from the saved offset, and drains the backlog without a visible gap larger than the outage itself.

Practitioner takeaway: Crash recovery is reliable only when persistence, checkpointing, and downstream retry behaviour are aligned, because the collector can resume state but it cannot guarantee perfect end-to-end delivery by itself.

Related resources from NHI Mgmt Group

• OpenTelemetry Collector
• OpenTelemetry
• What is the difference between a telemetry collector that only forwards OpenTelemetry data and one that also parses and reformats it?
• What happens when a user publishes a password video that shows entropy while the account is still reachable?