Use the Agent Gateway pattern so collectors near workloads forward telemetry to centralized gateway collectors. Run multiple gateway instances behind a load balancer, give each instance separate persistent storage for its queue, and keep batching, retries, and failover enabled. This reduces data loss during node failures, rolling updates, and backend outages while preserving horizontal scalability.
How to structure an OpenTelemetry Collector HA design
High availability for the OpenTelemetry Collector works best when you treat it as a tiered telemetry pipeline, not a single daemon. Keep collectors close to workloads for collection and light processing, then forward to a resilient gateway layer that can absorb node loss, maintenance, and transient backend failures without breaking ingestion.
The design goal is continuity of telemetry under failure, not just uptime of one process. That means you need redundancy at the gateway tier, buffering that survives restarts, and routing that lets collectors keep sending while an instance or backend is temporarily unavailable. The Observability, telemetry and SRE for security teams guide is useful here because it covers OpenTelemetry Collector hardening, telemetry pipeline security, and SLO-oriented alerting.
In practice, this architecture also fits a broader cloud design pattern: multiple instances behind a load balancer, each able to take traffic independently. That lets you scale the collector tier horizontally while avoiding a single point of failure in ingestion, and the Cloud Security Architecture Review Template is a useful companion when you want to review the design as a production workload rather than as a one-off service.
Queue durability matters as much as process redundancy. If a gateway collector buffers telemetry in memory only, a crash or reschedule can drop data even though the service looks healthy from the outside. The Security Data Pipeline Guide is relevant because it addresses collection, routing, filtering, and pipeline resilience as one operational flow, which is exactly how production telemetry should be designed.
What failure modes HA needs to absorb
Collector HA should be designed around the most common operational failures: a gateway pod restart, a node drain, a load balancer target becoming unhealthy, or the downstream analytics backend slowing down. Batching and retry logic should smooth transient interruptions, while per-instance persistent storage should protect queued data during restarts and rolling deploys.
The main mistake is assuming that “multiple replicas” alone is enough. If all replicas share the same failure domain, or if every replica writes to ephemeral storage only, you have availability at the service layer but not at the data layer. The Vulnerability Management for Network Devices Guide reinforces a similar production lesson: high availability has to be paired with lifecycle-safe upgrades and recovery planning, not just extra instances.
Another failure mode is backpressure during backend outages. When the destination cannot ingest fast enough, the collector must not collapse into data loss or unbounded resource use. A resilient design keeps retry windows bounded, preserves queue state, and makes overload visible so operators can distinguish a slow backend from a broken pipeline.
What production teams should verify before go-live
Before relying on the design, validate that failover works at each layer: collector replica failure, queue persistence after restart, load balancer rerouting, and backend recovery after an outage. The point is to prove that telemetry survives the exact incidents that are most likely in production, not just that the deployment is healthy under normal load.
It also helps to check whether the collector configuration itself is repeatable and reviewable. A hardened telemetry pipeline should have versioned config, clear ownership, and a release process that treats pipeline changes as production changes. The Security Architecture Review Guide is a good match for that review because it focuses on design changes, trust boundaries, and evidence needed for an architecture decision.
If the collector is also filtering, transforming, or redacting telemetry, verify that those functions still behave correctly after failover. HA should preserve both delivery and semantics, because a resilient pipeline that changes what gets emitted is still a broken pipeline from an operations standpoint.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST CSF 2.0 and NIST SP 800-53 Rev 5 set the technical controls, while ISO/IEC 27001:2022 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.IR-02 — Redundancy | Collector HA depends on redundant telemetry paths and components. |
| RC.RP-01 — Recovery Plan Execution | HA for collectors is validated by recovery after crashes, rollouts, and outages. | |
| Recommendation — Design redundant collector and backend paths so telemetry continues during node or service failure. Test recovery of collector queues and routing after failure events. | ||
| NIST SP 800-53 Rev 5 | SC-5 — Denial of Service Protection | Telemetry backpressure and backend outages can exhaust collector capacity. |
| CP-10 — System Recovery and Reconstitution | Persistent queues and restart behaviour are central to collector resilience. | |
| Recommendation — Configure buffering and retries to absorb transient overload without dropping telemetry. Persist collector state so telemetry can be restored after restart or failover. | ||
| ISO/IEC 27001:2022 | A.8.14 — Redundancy of information processing facilities | High availability for collectors requires redundant processing and failover design. |
| Recommendation — Provide redundant collector instances and failover paths for telemetry processing. | ||
Practitioner Guidance
What to prioritise: Treat persistent queue storage and backend retry behaviour as first-class production requirements, not optional tuning. If either one is missing, the deployment may be redundant in name only.
What to verify: Test the exact recovery path you expect in an incident, including pod eviction, node loss, and backend unavailability. You want evidence that queued telemetry is preserved, drains cleanly, and does not duplicate or stall after recovery.
Common mistake: Teams often add replicas but leave the queue state tied to ephemeral disks or assume the load balancer is the HA solution. The collector tier is only resilient when the buffering layer is resilient too.
Practitioner takeaway: For production observability, HA is not just about keeping the collector process up, it is about keeping telemetry flowing, durable, and recoverable across the failures you actually expect.
Related resources from NHI Mgmt Group
- How should security teams design syslog for high-availability logging?
- What breaks when OpenTelemetry Collector high availability is not configured correctly?
- How should security teams design workload IAM platforms for high availability across regions and availability zones?
- How should security teams design cloud identity platforms to maintain high availability during outages and maintenance?
Deepen Your Knowledge
Free weekly newsletter
Subscribe to the NHI & AI Identity Journal
The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.
Bonus 33% off our NHI Course when you subscribe.
Reviewed and updated by the NHIMG editorial team on November 5, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org