Notifications
Clear all
Topic starter
19/08/2026 3:30 pm
TL;DR: Production OpenTelemetry Collector pipelines are governed by processor order, sampling, enrichment, and backpressure decisions that determine cost, reliability, and whether critical traces survive, according to Sawmills. The real risk is not collector failure but misconfigured policy, where defaults quietly shape telemetry quality and spend.
NHIMG editorial — based on content published by Sawmills: The OpenTelemetry Collector in production, the processor pipeline that decides what your backend ever sees
Questions worth separating out
Q: How should teams govern custom OpenTelemetry collector builds in production?
A: Treat the manifest, build workflow, and release artifact as one control chain.
Q: When does telemetry filtering fail in practice?
A: Filtering fails when teams try to drop noise before the metadata needed to identify it has been attached, or when the pipeline is scaled without preserving trace locality.
Q: How do you know if collector sampling is actually working?
A: Check whether slow requests, error traces, and high-value exemplars still reach the backend while routine traffic drops as intended.
Practitioner guidance
- Review processor order as a control design Validate that memory protection runs before enrichment, filtering runs after required context is attached, and batching is last.
- Define explicit drop rules for noise sources Remove routine health checks, readiness probes, and other predictable chatter at the collector so they never reach the backend.
- Cap metric cardinality at the collector Delete or normalise high-cardinality attributes such as request identifiers and user-scoped keys before they become billable time series.
What's in the full article
Sawmills' full article covers the operational configuration detail this post intentionally leaves at the governance level:
- Pipeline examples showing exactly how processors are ordered for traces, metrics, and logs in production.
- Specific configuration snippets for memory limiting, filtering, transform rules, and tail sampling.
- Deployment guidance for agent, gateway, and sidecar collector patterns under real load.
- Operational examples of queue behaviour, backpressure handling, and scaling triggers.
👉 Read Sawmills' analysis of OpenTelemetry Collector processor governance in production →
OpenTelemetry Collector in production: are your controls keeping up?
Explore further
19/08/2026 4:13 pm
Telemetry governance is a policy problem, not a transport problem. The collector succeeds or fails based on how precisely teams define what to keep, drop, enrich, and sample. That is the same control logic identity teams face with access reviews and lifecycle enforcement: rules only work when they sit before sprawl and cost accumulation. The practitioner conclusion is to manage observability as governed policy, not passive plumbing.
A question worth separating out:
Q: What should teams do when collector backpressure builds during an incident?
A: Prioritise durable handling for audit logs and error traces, then allow lower-value debug telemetry to degrade first. If all signals share one queue policy, the collector will either lose important evidence or spend itself into instability. Differentiate by signal criticality before the next outage tests that assumption.
👉 Read our full editorial: OpenTelemetry Collector production controls decide what your backend sees
