Pipeline-level telemetry filtering cuts Splunk overage costs

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Apica's blog post: Telemetry sprawl and Splunk pricing complexity are creating avoidable cost overruns, and pipeline-level filtering is the lever that changes the economics.

Why it matters: IAM and security teams increasingly depend on telemetry pipelines to preserve signal quality while controlling spend, especially when logs, traces and metrics are treated as a governed data asset rather than a raw feed.

By the numbers:

• Apica says telemetry filtering can reduce Splunk ingest by 25–60% depending on the environment.
• Apica says one outcome is up to 40% reduction in Splunk ingest.
• Apica says Splunk invoices can grow 25–40% annually before optimisation.

Context

Telemetry cost sprawl happens when every source ships logs, metrics and traces directly into an analytics platform with no intermediate governance layer. In that model, the organisation pays for noise, duplication and verbose fields even when only a fraction of the data is operationally useful.

The problem is not limited to observability budgets. When security instrumentation and application telemetry are unmanaged, teams lose control over retention, routing and enrichment decisions that should be made before data becomes expensive to store and search.

In environments that combine cloud workloads, microservices and security monitoring, this creates a governance gap rather than a tooling gap: the issue is who decides what telemetry is worth keeping, not whether the platform can ingest it.

Key questions

Q: How do teams reduce observability ingest costs without losing signal quality?

A: Teams reduce observability ingest costs by filtering at the collector, dropping low-value events, sampling high-volume traces, and normalising data before storage. Common controls include include/exclude rules, metric aggregation, log severity thresholds, and route-by-label policies. The key is to preserve error, latency, and audit signals while suppressing duplicate, verbose, or transient noise.

Q: What is the difference between telemetry filtering and telemetry routing?

A: Telemetry filtering decides whether a data item is kept, sampled, or dropped before it is stored or forwarded. Telemetry routing decides where the kept data goes, based on rules such as source, service, severity, tenant, or compliance class. Filtering changes volume and content. Routing changes destination and retention path.

Q: How does OpenTelemetry help control vendor lock-in in observability pipelines?

A: OpenTelemetry helps control vendor lock-in by using standard instrumentation, semantic conventions, and collector pipelines that can export the same telemetry to multiple back ends. Teams keep collection, enrichment, sampling, and routing in portable components rather than in a proprietary agent or storage format. That makes switching vendors less disruptive to data collection and access control.

Q: When should organisations send telemetry to cheaper storage instead of a SIEM?

A: Organisations send telemetry to cheaper storage when the data has low real-time value, long retention needs, or is mainly used for later investigation rather than active alerting. Security logs that support threat hunting often stay in a SIEM, while verbose application traces, debug logs, and routine metrics can move to lower-cost object storage after filtering and indexing.

Technical breakdown

How telemetry pipelines reduce ingest before indexing

A telemetry pipeline sits between producers and backends, so it can reshape data before the destination platform charges for storage, compute or search. The key mechanism is pre-index filtering: dropping duplicate lines, excluding low-value fields, downsampling high-volume metrics and normalising verbose payloads into compact events. This is different from post-ingest cleanup, because the cost is avoided before indexing begins. In practice, the pipeline becomes a policy enforcement point for data volume, structure and destination choice.

Practical implication: Move filtering and enrichment upstream so expensive platforms receive only telemetry that is worth indexing.

Why raw telemetry creates cost and governance drift

Raw telemetry is expensive because modern environments generate multiple data classes at high volume, often with overlapping content. Developers add debug logs, security teams add more instrumentation, and microservices emit repeated events across layers. Without central routing and normalisation, each team optimises locally while the organisation absorbs the full ingest burden. The result is not just overspending, but inconsistent retention, unclear ownership and no shared decision model for what should be indexed, archived or discarded.

Practical implication: Treat telemetry volume as a governed resource with clear ownership, routing rules and retention policy.

OpenTelemetry and backend routing as architecture choices

OpenTelemetry matters here because it separates instrumentation from destination choice. That allows a single telemetry source to feed multiple backends, which reduces lock-in and makes offloading possible when a premium analytics platform is not the right place for every signal. Routing is the architectural lever that lets security, observability and data teams assign different value tiers to different streams. The important detail is that standardisation happens at collection time, while backend selection remains a policy decision.

Practical implication: Use standard telemetry instrumentation so routing decisions stay flexible as cost and data priorities change.

NHI Mgmt Group analysis

Telemetry governance is now a budget-control problem, not just an observability problem. When pipelines feed premium analytics platforms without intermediate policy, organisations pay for duplicate, low-value and over-verbose data at scale. That shifts telemetry from an engineering detail into a governance domain where finance, security and platform owners need a shared control point. Practitioners should treat ingestion policy as part of the security architecture, not as a downstream optimisation task.

Cost exposure grows fastest where telemetry lacks classification. If logs, metrics and traces all follow the same path, the platform absorbs everything at the highest-cost destination. That is especially important in security programmes where instrumentation expands quickly and nobody revisits whether the same data belongs in Splunk, object storage or a cheaper analytics layer. Practitioners should classify telemetry by business value before routing decisions are made.

Control over telemetry flow is the named concept that matters here: data-path governance. This is the discipline of deciding, before ingestion, which signals are retained, transformed, downsampled or redirected to a lower-cost backend. It becomes a practical control when organisations want both visibility and cost discipline without letting every team make independent shipping decisions. Practitioners should build policy around the data path, not around the destination alone.

Open standards reduce switching risk, but they do not remove governance responsibility. Standardised collection makes it easier to move telemetry between backends and to avoid tool-specific instrumentation debt. The harder part remains policy design: which signals deserve premium search, which should be archived, and which should never reach the expensive platform in the first place. Practitioners should use standards to preserve optionality, then enforce routing governance on top.

Security telemetry and operational telemetry should not be priced or governed identically. The article points to a common failure mode where all data is treated as equally valuable simply because it is available. That assumption drives overspend and noise, especially when teams fear losing visibility. Practitioners should create separate value tiers for security-critical, investigative and routine telemetry so cost decisions match operational need.

What this signals

Data-path governance will become a standard control in high-volume observability environments. As telemetry volumes grow, the control point shifts from backend tuning to upstream policy about what should be kept, transformed or redirected. Teams that cannot govern the data path will keep paying for every source that emits too much, too often.

Cost pressure will force security teams to defend telemetry value in operational terms. Visibility alone will no longer justify unlimited ingest when the same signals can often be filtered, compressed or rerouted. That means security leaders will need to tie each telemetry class to a clear investigative or detection use case before it enters a premium platform.

For practitioners

• Classify telemetry by business value before ingestion Define which log, metric and trace streams must land in premium analytics, which can be transformed, and which should be routed elsewhere.
• Apply pre-index filtering to high-volume sources Drop duplicate events, trim verbose fields, downsample noisy metrics and suppress low-value debug output before data reaches the backend.
• Separate security telemetry from routine telemetry Create routing rules that preserve investigative signals in Splunk while sending lower-value streams to object storage or cheaper tools.
• Standardise collection across teams Use one instrumentation model so different teams do not create unmanaged telemetry paths, duplicate ingest or inconsistent retention practices.

Key takeaways

• Uncontrolled telemetry growth can turn observability platforms into cost multipliers when every source ships raw data directly into premium analytics.
• Pipeline-level filtering, routing and normalisation give organisations a way to preserve useful signal while cutting unnecessary ingest and storage.
• The governance question is no longer whether telemetry exists, but which signals deserve the expensive path and who decides that policy.

Related NHI Mgmt Group resources: Security Metrics and KPIs Programme Guide · Cloud Workload Identity Buyer's Guide · NetFlow and IPFIX Flow Telemetry Guide · Splunk Enterprise Security Guide

Key terms

• Telemetry pipeline: A telemetry pipeline is the path security data follows from collection to analysis and retention. In mature environments it must preserve context, maintain throughput, and avoid introducing blind spots as sources, formats, and volumes change over time.
• Pre-Index Filtering: Pre-Index Filtering is the removal of telemetry data before it is written into a search index or analytics store. It applies in log pipelines, metrics collectors, and tracing systems, where rules can drop noise, exclude fields, or sample events before indexing. That cuts storage and query load.
• OpenTelemetry: An open standard for collecting and moving traces, metrics, and logs in a consistent format. It helps teams avoid re-instrumenting every system when tools change, and it supports cross-platform analysis by keeping telemetry structured and portable.
• Telemetry routing: The process of directing logs and events to different destinations based on value, urgency, and investigative need. In mature SOC designs, routing is a security control because it determines which events remain searchable in the SIEM and which are archived more cheaply.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore nhimg.org for resources that connect identity governance to the broader security disciplines your programme depends on.