Telemetry pipelines are the missing control layer for observability costs

At a glance

What this is: Sawmills frames telemetry pipelines as the control layer that determines which observability data is kept, reshaped, routed, or discarded before backend costs are incurred.

Why it matters: For platform and security teams, this matters because pipeline policy governs cost, reliability, and signal quality before telemetry reaches the systems that store and query it.

Context

A telemetry pipeline is the governance layer between producers of logs, metrics, and traces and the observability backend that stores them. The cost problem is not usually the backend itself, but the upstream choice of what to collect, transform, sample, and drop before ingestion turns raw telemetry into billable volume.

For teams running OpenTelemetry, the important distinction is that the Collector executes policy, it does not define it. That makes telemetry governance a continuous operating problem, especially in Kubernetes and other fast-changing environments where instrumentation drift, high-cardinality labels, and noisy defaults quickly erode both efficiency and signal quality.

Key questions

Q: What breaks when telemetry pipelines are not treated as sensitive data flows?

A: Secrets, tokens, and identity context can replicate across brokers, caches, and SIEM systems before redaction occurs, which means attackers may harvest usable credentials long before defenders notice. The failure is not just leakage. It is persistence of exposed identity material across multiple hops, which expands the blast radius and complicates containment.

Q: Why do high-cardinality metrics cause so many observability problems?

A: High-cardinality metrics create too many unique time series, which raises cost, slows queries, and can make dashboards unusable during incidents. They are especially risky when agents or hurried developers use request-specific values as labels. Metrics should aggregate behaviour, while traces and logs carry request-level context under tighter control.

Q: How do teams know if telemetry sampling is working properly?

A: A sampling policy is working when routine traffic falls away but errors, slow traces, and incident-relevant signals remain visible. If dashboards still show the same noise, or if investigations lack the traces that matter, sampling is either too shallow or placed at the wrong stage of the pipeline.

Q: When should observability teams centralise processing instead of keeping it on the node?

A: Centralise the decisions that require a global view, especially tail sampling and cross-service routing, and keep node-local processing for log tailing, redaction, and first-pass filtering. If the decision depends on the whole trace, it belongs where the whole trace can still be assembled.

Technical breakdown

Ingest, process, route: where telemetry pipelines decide cost

A telemetry pipeline is not a passive relay. It is a processing chain that receives telemetry from sources, applies in-memory logic, and exports selected data to one or more backends. Receivers ingest logs, metrics, and traces, processors filter and reshape them, and exporters send the result onward. The architectural mistake is to treat this as plumbing and push all control into the observability platform, where ingestion, parsing, indexing, and storage costs are already locked in. The middle stage, processing, is where cost, fidelity, and usefulness are actually negotiated.

Practical implication: Design telemetry pipelines as policy engines, not forwarding paths, so the control point sits before ingestion billing begins.

Cardinality, filtering, and tail sampling are the main cost levers

Telemetry cost rises fastest when teams fail to constrain unnecessary data at source. Filtering removes known-noise signals such as health checks. Transforming attributes prevents high-cardinality labels like user_id, pod_name, and request_id from exploding one metric into thousands of series. Tail sampling is especially important for traces because it decides after a trace completes, which means errors and slow requests can be kept while routine traffic is sampled down. These controls matter most in ephemeral environments, where infrastructure churn can create cost growth without any corresponding increase in diagnostic value.

Practical implication: Strip high-cardinality attributes and apply tail sampling where trace completeness matters more than uniform capture.

Gateway and agent placement determines what can be controlled safely

Where the pipeline runs changes what it can do well. Node-local agents are best for low-value work such as log tailing, first-pass filtering, redaction, and buffering. Central gateways are better for decisions that require a global view, especially tail sampling and cross-service routing. The ordering of processors also matters because memory protection, enrichment, filtering, transformation, and batching each affect the next stage. If teams place the wrong logic at the wrong tier, they either waste bandwidth and compute or lose the ability to make globally consistent decisions about what telemetry should survive.

Practical implication: Split node-local and central processing deliberately, and place sampling decisions where the full trace can still be evaluated.

NHI Mgmt Group analysis

Telemetry governance is becoming a control plane problem, not a tooling problem. The article shows that cost is decided upstream of the observability backend, where policy still exists and waste can still be removed. That shifts the real question from which platform to buy to which processing decisions deserve governance. Practitioners should treat telemetry policy as a managed control layer, not a by-product of instrumentation.

Cardinality sprawl is the observability equivalent of identity sprawl. When a small set of attributes turns one metric into millions of time series, the organisation has lost control of scope. The parallel to identity governance is direct: unmanaged dimensions create cost and operational noise just as unmanaged entitlements create risk. Telemetry cardinality debt: the accumulated cost and instability created when teams allow telemetry dimensions to multiply without governance. Practitioners should identify which attributes are actually decision-bearing and suppress the rest.

Tail sampling changes the economics of trace retention. Keeping only traces that contain errors or latency spikes is more defensible than capturing everything at full fidelity, but only if the sampling decision happens after the trace completes. That makes gateway placement a governance choice, not just a deployment pattern. Teams should align retention decisions with business value, incident response needs, and backend cost exposure.

Observability cost control fails when no one owns the source of telemetry waste. The article describes a structural gap between the teams generating telemetry and the teams paying for it. That gap is common across cloud, IAM, and NHI governance programmes: downstream consumers inherit the cost of upstream sprawl. Practitioners should assign explicit ownership for telemetry policy and review it as part of platform governance.

What this signals

Telemetry governance now looks more like policy engineering than infrastructure wiring, and that matters because the cost of poor control shows up long before anyone notices a backend bill. The practical shift is to manage what telemetry exists, not just where it lands.

Telemetry cardinality debt: once teams let unstable labels and noisy defaults proliferate, observability costs become structurally harder to reverse. The control point has to move upstream into collection policy, sampling discipline, and attribute governance.

For practitioners

• Define a telemetry policy layer Separate collection from control by documenting what must be kept, what can be filtered, and what must be sampled before backend ingestion.
• Remove high-cardinality attributes Audit metrics for dimensions such as user_id, pod_name, and request_id, then delete or reshape them before they reach storage and indexing.
• Use tail sampling for traces Keep errors and slow requests with post-completion trace sampling, and avoid head-based defaults when latency or failure context matters.
• Place processing at the right tier Run node-local agents for log tailing, redaction, and first-pass filtering, then use a central gateway for global routing and sampling decisions.

Key takeaways

• Telemetry pipelines are the place where observability cost, signal quality, and reliability are actually decided.
• Filtering, attribute shaping, and tail sampling do most of the financial work because they reduce waste before ingestion charges accumulate.
• Teams that want predictable observability spend need explicit ownership of telemetry policy, not just a better backend contract.

Key terms

• Telemetry pipeline: A telemetry pipeline is the path security data follows from collection to analysis and retention. In mature environments it must preserve context, maintain throughput, and avoid introducing blind spots as sources, formats, and volumes change over time.
• Cardinality: The number of unique attribute combinations in telemetry data. High cardinality increases the number of metric series and can inflate cost, overwhelm backends, and make analysis noisy unless attributes are normalised or removed before export.
• Trace Sampling: Trace sampling is the practice of recording only a subset of traces to control overhead and scale observability systems. It reduces volume, but it can also obscure rare failures if context is not preserved consistently. Good sampling design balances cost with enough fidelity to diagnose authentication and routing issues.
• Observability Cost Governance: Observability cost governance is the practice of controlling what telemetry is collected, retained, and routed so the organisation pays only for data that has diagnostic value. It sits upstream of the backend and is most effective when owned as policy, not as an afterthought.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore nhimg.org for resources that connect identity governance to the broader security disciplines your programme depends on.