TL;DR: Exploding observability costs, inconsistent data quality, and brittle OpenTelemetry pipelines that break under change define a telemetry problem in KubeCon North America conversations, according to Sawmills. The practical implication is that teams need upstream control over telemetry volume, schema quality, and pipeline reliability before cost and signal loss become structural.
At a glance
What this is: Sawmills argues that observability pain is now defined by cost pressure, noisy data, and brittle OpenTelemetry pipelines that cannot absorb change cleanly.
Why it matters: This matters because telemetry pipelines are part of the operational control plane, and when they are fragile, teams lose both reliable visibility and the ability to govern cost or quality.
Context
Telemetry is only useful when it is both affordable and trustworthy enough to support troubleshooting, automation, and security operations. When logs, metrics, and traces arrive in inconsistent shapes or at excessive volume, teams end up paying to store and process data that does not improve decisions. The operational problem is not just observability spend, but the lack of control over what enters the pipeline and how it changes over time.
In cloud-native environments, OpenTelemetry pipelines sit between production systems and the tools that consume telemetry, so a restart, rollout, or schema mismatch can create blind spots. That makes pipeline governance relevant to broader security and resilience work as well, because data loss and broken context affect incident response, service reliability, and any control that depends on continuous signal flow.
Key questions
Q: How should teams reduce observability costs without losing useful telemetry?
A: Start at the pipeline, not the backend. Filter obvious noise, remove high-cardinality fields, and tail-sample traces so routine traffic is reduced after full context is available. That preserves error and latency signal while cutting ingest, storage, and query costs before they become locked into the observability bill.
Q: Why do telemetry pipelines fail when teams make small configuration changes?
A: They fail because the pipeline often behaves like a deployable component rather than a continuously running control layer. If a change requires restarts or rollouts, even minor edits can interrupt data flow, create gaps, or produce inconsistent transformation results. The operational risk is the interruption itself, not just the updated configuration.
Q: What signals show that data governance is not actually working?
A: Common warning signs are repeated manual rework, conflicting metrics across departments, slow approvals, and frequent disputes about what a data element means. If visibility exists but decisions are still inconsistent, governance is producing reporting rather than control. That is a sign the model needs tighter ownership and lineage.
Q: What should teams do when telemetry changes risk interrupting production data flow?
A: Treat the change as a continuity problem and require a reconfiguration path that preserves live data flow. The goal is to update processors, filters, samplers, and transformers without breaking the collector’s ability to receive and forward telemetry. If that is not possible, the pipeline is too brittle for the environment it supports.
Technical breakdown
Why telemetry pipelines become brittle under change
OpenTelemetry Collector pipelines are made up of processors, filters, samplers, exporters, and transformers that shape data before it reaches downstream tools. The fragility appears when configuration changes require a restart or rollout, because the pipeline stops behaving like an always-on data plane and starts behaving like a deployment artifact. That creates a narrow window where telemetry can be delayed, dropped, or reshaped inconsistently. In practice, teams are not only managing observability content, but also the reliability of the transport and transformation layer that makes the content usable.
Practical implication: Treat telemetry configuration changes as availability-sensitive operations, not routine edits.
How telemetry waste turns into cost and governance debt
Telemetry waste usually starts with over-collection: unnecessary logs, excessive metrics, and traces with high cardinality or poor filtering. Once that data reaches an observability platform, the cost compounds because teams pay to ingest, store, index, and query information that adds little diagnostic value. Governance debt follows when schema quality is weak, because unstructured logs and missing attributes make the data harder to use for automation or correlation. The result is a control problem, not just a spending problem: teams cannot easily decide what should be kept, dropped, enriched, or normalized.
Practical implication: Put volume controls and schema validation upstream of storage and analysis.
What hot reload changes in pipeline operations
Hot reload shifts pipeline changes from disruptive redeployments to in-place configuration updates. That matters because the operational risk is not the feature itself, but the interruption caused by changing telemetry logic in a live environment. If processors, samplers, and filters can be updated without downtime, teams reduce the chance of data gaps and avoid forcing production changes into collector restarts. The architectural point is simple: telemetry control should be dynamic enough to evolve with the system it observes, otherwise teams inherit a permanent trade-off between change velocity and signal continuity.
Practical implication: Prefer live reconfiguration paths that preserve data flow while policies evolve.
Threat narrative
Attacker objective: There is no adversary objective in the article; the risk is accidental loss of telemetry continuity and control during pipeline changes.
- Entry occurs through routine pipeline change management, where a small configuration update can alter telemetry flow in production.
- Escalation happens when a collector restart or rollout interrupts data collection, creating gaps, blind spots, or broken context across dependent tools.
- Impact is operational rather than malicious: teams lose reliable observability, and troubleshooting, automation, and cost controls all degrade at once.
Breaches seen in the wild
- CI/CD pipeline exploitation case study: Credentials in an exposed .git/config let a researcher edit a Bitbucket pipeline so it planted their SSH key on the server. No victim was named.
- reviewdog Action compromise 2025: A stolen maintainer token poisoned reviewdog/action-setup, leaking CI secrets including the tj-actions bot token used in the next attack.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Telemetry sovereignty is becoming a governance problem, not a tooling preference. Once teams depend on upstream control over what gets collected, filtered, and transformed, observability stops being a passive feed and becomes an actively governed data product. That shifts the centre of gravity from dashboards to the pipeline itself, where reliability and policy enforcement have to be managed together. Practitioners should treat pipeline ownership as part of operational control, not just observability engineering.
Pipeline fragility is the hidden tax on cost optimisation. Cutting ingest volume without preserving schema quality and change resilience simply moves the pain from storage bills to operational disruption. The real failure mode is not high telemetry volume alone, but a pipeline that cannot adapt without interruption. That is why cost reduction, data quality, and reliability need to be handled as one governance problem.
Telemetry quality now sits on the same side of the risk ledger as uptime. When noisy or inconsistent data weakens automation, incident triage slows and control confidence falls. In a cloud-native environment, a brittle telemetry pipeline can be as damaging as a blind spot in detection, because both reduce the organisation's ability to see and act. Teams should measure telemetry health as a resilience indicator, not a back-office engineering metric.
AI-assisted telemetry management is a response to complexity, not a substitute for control design. Automating waste detection or schema fixes can help at scale, but only if the underlying policy boundaries are clear. The practical challenge is deciding what telemetry should exist, at what quality, and with what continuity guarantees. Practitioners should use automation to enforce governance, not to postpone it.
OpenTelemetry hot reload points to a broader architectural shift. The market is moving away from brittle, manually interrupted telemetry changes toward dynamically managed pipelines that can absorb policy updates in flight. That direction validates the need for continuous control of collection, transformation, and delivery, especially where observability underpins security and resilience operations. Teams should evaluate whether their current pipeline design can change without creating its own outage.
What this signals
Telemetry governance now belongs upstream of the observability platform. Teams that only optimise retention and dashboards will keep paying for waste that should have been removed before ingestion. The more resilient model is to treat telemetry policy, schema hygiene, and pipeline continuity as part of the same operating control.
Pipeline fragility is a resilience issue as much as a cost issue. When collector changes require restarts, the organisation inherits avoidable signal loss during normal operations. That makes change-safe telemetry architecture a practical requirement for anyone responsible for service reliability, incident response, or security monitoring.
For practitioners
- Define upstream telemetry policy controls Set explicit rules for what logs, metrics, and traces are collected, enriched, sampled, or dropped before data reaches storage.
- Instrument schema validation and quality checks Detect missing fields, inconsistent attributes, and noisy event patterns before they spread into downstream dashboards and automations.
- Reduce collector change blast radius Use configuration paths that support live updates for processors, filters, samplers, and transformers without forcing collector restarts.
- Track telemetry cost against value Review ingest, storage, and query spend alongside the troubleshooting value and automation utility of each telemetry class.
- Measure telemetry continuity as an operational control Monitor data gaps, dropped events, and pipeline interruptions as service-impacting issues rather than background engineering noise.
Key takeaways
- Observability problems are increasingly driven by excess telemetry, poor data quality, and fragile pipeline operations rather than by dashboards alone.
- Small collector changes can create outsized operational risk when they force restarts, rollouts, or data gaps in production.
- Teams should move control upstream so cost reduction, schema hygiene, and continuity are managed before telemetry reaches expensive downstream systems.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST CSF 2.0, NIST SP 800-53 Rev 5 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.DS-01 — Data-at-Rest Protection | Telemetry governance depends on protecting the integrity and usefulness of collected data. |
| PR.PS-01 — Configuration Management | Collector restarts and rollout risk show why telemetry pipelines need controlled change management. | |
| Recommendation — Apply data handling controls to keep telemetry accurate, usable, and governed before storage and analysis. Use configuration management to change telemetry pipelines without disrupting live data flow. | ||
| NIST SP 800-53 Rev 5 | CM-2 — Baseline Configuration | Pipeline fragility is fundamentally a configuration control problem in production. |
| Recommendation — Establish baselines for collector configurations and approve changes through controlled updates. | ||
| CIS Controls v8 | CIS-4 — Secure Configuration of Enterprise Assets and Software | Telemetry collectors and pipelines need hardened, repeatable configuration to avoid brittle changes. |
| Recommendation — Harden collector and pipeline configurations so routine edits do not create outages or data loss. | ||
Key terms
- Telemetry pipeline: A telemetry pipeline is the path security data follows from collection to analysis and retention. In mature environments it must preserve context, maintain throughput, and avoid introducing blind spots as sources, formats, and volumes change over time.
- Telemetry-driven governance: Telemetry-driven governance is a control approach that relies on runtime signals rather than periodic paperwork. For AI, that means watching drift, leakage, prompt anomalies, and other live indicators so governance decisions reflect current system behaviour instead of stale review findings.
- Schema Quality: The degree to which telemetry data arrives with consistent structure, required attributes, and usable context. Weak schema quality makes logs harder to search, correlate, and automate against, and it often turns observability spend into noise instead of decision support.
- Routing Fragility: The gap between a model that appears capable in general and one that is reliable enough for a specific workflow step. It becomes visible when exact output, timing, or completion requirements are stricter than the model’s average benchmark performance.
Deepen your knowledge
NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is the industry's only accredited NHI security programme, designed for practitioners who need a structured way to govern identities, access, and secrets.
Published by the NHIMG editorial team on August 19, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org