Notifications
Clear all
Topic starter
19/08/2026 3:59 pm
TL;DR: Telemetry can become an outage amplifier when observability agents overwhelm control planes, as Sawmills describes alongside OpenAI’s Kubernetes incident. The operational lesson is that monitoring systems need explicit resource limits, rate controls, and isolated failure domains before visibility can be trusted at scale.
NHIMG editorial — based on content published by Sawmills: Managing telemetry deployments while safeguarding infrastructure
Questions worth separating out
Q: How should teams prevent telemetry systems from taking down production services?
A: Teams should treat telemetry as a bounded workload with explicit limits, separate failure domains, and staged rollout controls.
Q: When does visibility tooling create more risk than it reduces?
A: Visibility tooling creates more risk when it requires inbound access, shared credentials, or broad firewall changes just to collect data.
Q: What do security teams get wrong about telemetry governance?
A: They often assume observability is low-risk infrastructure because it exists to help operations.
Practitioner guidance
- Implement hard resource caps for observability agents Set CPU, memory, and bandwidth ceilings for collectors and exporters so telemetry cannot consume shared production capacity during spikes.
- Isolate telemetry control planes from production workloads Run ingestion, buffering, and storage on separate infrastructure with dedicated scaling policies so failures in monitoring do not cascade into customer-facing services.
- Introduce rate limits and adaptive sampling Throttle high-volume sources, reduce low-value event noise, and increase sampling during surges so observability remains bounded under stress.
What's in the full article
Sawmills' full article covers the operational detail this post intentionally leaves for the source:
- Concrete Datadog-style agent limit examples for CPU and memory tuning in telemetry configurations
- Specific alert thresholds for ingestion-rate anomalies, buffer depth, pipeline latency, and storage growth
- Practical patterns for spike protection, including intelligent routing, shaping, and filtering of high-volume data
- Implementation detail on tenant-level separation and isolated control planes for observability resilience
👉 Read Sawmills' analysis of telemetry guardrails and observability outages →
Telemetry guardrails and observability outages: what teams are missing?
Explore further
19/08/2026 4:36 pm
Telemetry guardrails are a reliability control, not a tuning preference. When observability tooling can consume control-plane resources without hard limits, it becomes part of the outage path. The article shows that resource caps, rate limits, and circuit breakers are not optional optimisation features; they are the controls that separate useful monitoring from systemic instability. Practitioners should treat telemetry governance as part of production resilience.
A question worth separating out:
Q: Who should own the risk when telemetry changes affect service availability?
A: Ownership should sit with the platform or SRE function, but security and identity teams should set the access and change-control expectations. If a telemetry change can interrupt production, it needs the same accountability model as other privileged infrastructure changes, including approval, testing, and rollback responsibility.
👉 Read our full editorial: Telemetry guardrails are now a production reliability control
