Upstream telemetry filtering turns observability cost into governance

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Apica's blog post: Upstream telemetry filtering reduces Datadog ingestion cost by stopping noisy logs before they are billed and stored.

Why it matters: Identity, cloud, and SOC teams need pipeline-level control because downstream observability tools cannot recover cost, privacy, or signal quality once telemetry has already been ingested.

By the numbers:

• Apica customers who apply pipeline filtering consistently see observability spending drop by 40%.

Context

Telemetry pipelines control what data moves from sources into observability and security platforms. When those pipelines are absent or weak, teams tend to forward everything, then pay to store, search, and alert on data that never needed to arrive in the first place.

The governance gap is not collection itself but choice at ingestion time. Security data pipeline controls matter because upstream filtering, routing, and reduction determine whether telemetry becomes a managed asset or an expensive exhaust stream.

In environments with large log estates, the starting position is usually reactive rather than deliberate: teams add sources faster than they define rules for retention, filtering, and forwarding.

Key questions

Q: How does a telemetry pipeline reduce observability spend?

A: A telemetry pipeline reduces observability spend by shaping data before it reaches expensive storage and query layers. It can drop noisy fields, sample high-volume traces, aggregate metrics, compress payloads, and route different data classes to cheaper retention tiers. The main savings come from less ingestion volume, shorter retention, and fewer costly searches.

Q: Why does sending every log to an observability platform create waste?

A: Sending every log to an observability platform creates waste when low-value, duplicate or noisy events are stored, indexed and queried at full cost. Excess ingestion increases storage, licence and search overhead, and it can bury security-relevant events such as authentication failures, privilege changes or error bursts. Tight source-level collection rules, severity thresholds and parsing filters reduce that load.

Q: How can upstream filtering improve log quality?

A: Upstream filtering improves log quality by dropping duplicate, debug-level or malformed records before they consume ingestion, indexing and retention capacity. Applied at the source or collector, it can normalise fields, redact sensitive values and preserve a cleaner evidence set for incident response, detection engineering and compliance review.

Q: When should teams filter telemetry before ingestion?

A: Teams should filter telemetry before ingestion when the source emits high-volume noise, when only specific event classes are needed for detection or audit, or when storage and query costs are out of proportion to the data's value. Doing so at the agent, forwarder or collector limits duplicate events, malformed records and unnecessary sensitive data exposure.

Technical breakdown

How upstream log filtering changes ingestion economics

A telemetry pipeline sits between producers and downstream platforms, intercepting events before they are stored or indexed. Filtering at this stage removes labels, message types, or whole classes of logs before they count toward billable volume. The technical distinction matters because post-ingestion filtering still pays the storage and indexing cost. Upstream filtering shifts the decision point left, so the control is applied while data is still in motion rather than after the bill has already been incurred. See the ETL and ELT Pipeline Security Guide.

Practical implication: Define filtering rules before ingestion so cost, retention, and exposure are controlled at the point of collection.

Why noisy telemetry weakens observability quality

High-volume telemetry is not automatically better telemetry. When logs contain repetitive debug events, unused labels, or low-value service output, the signal-to-noise ratio drops across dashboards, alerts, and investigations. Operators then tune around irrelevant data, which increases false positives and slows root cause analysis. A pipeline that reduces noise improves downstream detection quality because it changes the evidence set the observability tool receives, not just the amount of data it stores. Read Observability, telemetry and SRE for security teams.

Practical implication: Remove low-value events upstream so alert tuning and investigation workflows operate on data that is actually actionable.

How pipeline governance affects telemetry sovereignty

Once telemetry is forwarded into a third-party platform, organisations often lose practical visibility into which data left the environment and under what policy. That becomes a governance problem when logs can contain sensitive operational detail, identifiers, or regulated data. A controlled pipeline creates a checkpoint for classification, reduction, and routing, which is especially important where internal policy or regulatory scope limits what may be exported. The architecture therefore functions as a governance boundary, not just a transport layer. See the Security Data Pipeline Guide.

Practical implication: Treat telemetry pipelines as policy enforcement points for data minimisation and export control.

Breaches seen in the wild

• ArtiPACKED 2024: Unit 42 found public GitHub Actions artifacts in major open source projects leaking live GITHUB_TOKENs, runtime tokens and cloud tokens.
• reviewdog Action compromise 2025: A stolen maintainer token poisoned reviewdog/action-setup, leaking CI secrets including the tj-actions bot token used in the next attack.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Pipeline control is now a governance control, not an optimisation accessory. Once telemetry volumes grow across services and platforms, the decision about what gets ingested becomes a policy decision about cost, retention, and exposure. Teams that leave that choice to downstream tools are not just overspending, they are surrendering control over what enters their observability estate. The practitioner takeaway is to govern telemetry before it becomes billable data.

Upstream filtering creates a named governance pattern: telemetry ingestion discipline. The value lies in making collection intentional, with rules that separate useful evidence from operational noise before the platform stores either. That pattern matters because every added integration expands the noise surface as well as the cost base. The practitioner conclusion is that observability scale without pipeline discipline becomes self-inflicted complexity.

Data sovereignty starts at the telemetry boundary. When logs cross into a third-party service without a clear pipeline layer, organisations lose a practical checkpoint for classification and minimisation. That is especially relevant where internal policy or regional obligations constrain data movement. The practitioner conclusion is to treat telemetry export as governed movement, not routine plumbing.

Cost control and signal quality are linked, not separate projects. The same filter rules that reduce billable volume also improve the quality of dashboards, alerts, and investigations. That means observability teams should evaluate pipeline policy alongside detection outcomes, because the cheapest data is not always the least useful, and the best signal usually begins before ingestion. The practitioner conclusion is to tune the pipeline and the platform together.

Identity and access governance extends into observability pipelines. The pipeline becomes the control point for who can route data, which sources can flow, and what fields are retained. In that sense it behaves like an access policy for telemetry, even when the subject is logs rather than user accounts. The practitioner conclusion is to assign ownership and review rights to the pipeline as part of the wider governance model.

What this signals

Telemetry pipelines are becoming the practical boundary between observability and data governance. As log estates expand, the harder problem is not collection but deciding which events deserve to cross into a paid, searchable system. Teams that do not define that boundary early end up tuning around their own noise instead of using telemetry as evidence.

Telemetery ingestion discipline will increasingly sit alongside cost and compliance review. The same pipeline rules that suppress waste also reduce unnecessary data movement, which matters when logs may contain operational or regulated information. Practitioners should expect pipeline policy to become part of observability design rather than an afterthought.

Observability quality improves when the pipeline is treated as a control plane. That means routing, dropping, and field reduction should be owned and reviewed with the same seriousness as access and retention decisions. The programme implication is straightforward: if you cannot explain what enters the platform, you do not yet govern the platform.

For practitioners

• Set ingestion rules before the first full rollout Start with one high-volume log source, define what should be dropped, and validate the reduction in a staging or preview view before expanding to more services.
• Prioritise the noisiest low-value sources first Target cart service logs, background jobs, verbose health checks, and low-utility Kafka events before touching streams that support active investigations.
• Strip unnecessary labels from structured logs Review fields added for old debugging needs and remove those labels upstream so they never count toward storage or ingestion costs.
• Separate debug noise from incident evidence Filter debug-level output from stable services where it does not contribute to alerting or root cause analysis, and keep only the events that support operational decisions.

Key takeaways

• Telemetry cost problems often start upstream, where teams forward more log data than they can reasonably use or justify.
• Filtering before ingestion reduces both spend and noise, which improves the quality of the signals downstream tools receive.
• Pipeline ownership turns observability into a governed process, making it easier to control what data crosses the platform boundary.

Related NHI Mgmt Group resources: Security Data Pipeline Guide · Observability, telemetry and SRE for security teams · ETL and ELT Pipeline Security Guide

Key terms

• Observability: Observability is the ability to understand the internal state of a system from the data it produces. In security and operations, that means combining logs, metrics, and traces so teams can explain why something happened, not just confirm that something changed.
• Ingestion Volume: Ingestion Volume is the amount of telemetry, logs or other data a platform accepts over a period. It usually refers to the rate and total size of incoming data, which affects storage, indexing, query performance and cost, especially when noisy sources send redundant or low-value events.
• Telemetry pipeline: A telemetry pipeline is the path security data follows from collection to analysis and retention. In mature environments it must preserve context, maintain throughput, and avoid introducing blind spots as sources, formats, and volumes change over time.
• Log Filtering: Log filtering is the practice of removing irrelevant or low-value records before or after they enter a security analytics platform. It reduces storage cost and alert noise, while preserving the events most likely to support detection, hunting, and response. Filtering can operate on raw events or normalized fields.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and workload identity. It gives identity and security practitioners a common vocabulary for governing access, secrets, and runtime trust.