Platform-level computation usually means customers pay twice, first to ingest and store logs, then again to store the derived metrics. That model also keeps large raw datasets in the platform even when only a few signals are needed. Edge processing reduces that duplication by compressing useful telemetry earlier, which can lower storage demand and make routing decisions more efficient.
Why platform-level metric computation drives up observability spend
Computing metrics inside the observability platform is expensive because the platform must retain and process the raw logs before it can derive the smaller metric set. That creates two billable layers of work, storage for the source telemetry and storage for the derived output, while also preserving data that may never be queried again. The cost problem is architectural, not just a pricing-model issue.
At a systems level, the platform becomes the place where high-volume telemetry is normalised, aggregated, indexed, and kept available for later analysis. That is useful when teams need flexible ad hoc investigation, but it means the platform must carry the full ingestion burden even when the business only needs a narrow set of signals. The more events, fields, and retention windows you keep, the more the cost curve is dominated by duplication rather than insight.
Another reason cost rises is that platform-side computation tends to happen after transport, so every event is moved, stored, and processed before any reduction occurs. If the metric can be created earlier, closer to the source or edge, the organisation can discard or compress some raw volume before it reaches the platform. That changes the economics by reducing both stored bytes and downstream compute demand, which is why edge processing often improves routing efficiency as well as cost control.
Where the duplication and retention burden shows up
The first cost driver is storage duplication. Raw logs remain in the platform for search, forensics, and retention policy, while the derived metrics are stored separately for dashboards and alerting. Even if the metric is small, the original records still consume ingestion, indexing, and retention capacity, so the platform is paying to preserve both the detailed record and the summarised view.
The second driver is compute amplification. Turning logs into metrics is not a passive transformation. It requires parsing, filtering, aggregation, enrichment, and often repeated recalculation when queries or rules change. That means the platform is not merely a repository, it is also an active processing layer whose cost grows with event volume and with the number of transformations applied to each event stream.
The third driver is retention mismatch. Metrics are usually kept because they are cheap to query and useful over time, but the raw logs that produced them are often retained for much longer than needed for the metric use case. When the raw dataset stays large, the derived metric does not replace the log cost, it sits on top of it. For teams operating at scale, that mismatch is often the real budget surprise.
Risk and Threat Considerations
When observability pipelines retain more raw telemetry than they need, cost becomes a governance issue as well as a storage issue. Excessive retention can also increase the exposure of sensitive operational data, because logs often contain identifiers, tokens, request payloads, or other material that should not be kept broadly or for longer than necessary.
Failure mechanism: Metrics are computed after full ingestion, so the platform accumulates raw events, derived metrics, and transformation work at the same time. If retention and access controls are not tightly bounded, organisations can end up paying for duplicated data while also expanding the blast radius of every stored log record.
Impact: Higher observability cost, slower pipelines, larger attack surface for exposed telemetry, and more difficulty proving that retention is proportionate to the monitoring objective.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
CIS Controls v8 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| CIS Controls v8 | 8 — Audit Log Management | Log volume and retention choices directly affect storage and monitoring cost. |
| 3 — Data Protection | Keeping raw logs longer than needed can expand exposure of sensitive telemetry. | |
| Recommendation — Tune log collection and retention to keep only the telemetry needed for detection and investigation. Classify telemetry and reduce retention for sensitive log data that no longer serves a security purpose. | ||
| NIST CSF 2.0 | GV.1 — Organizational Context | Observability retention should align with business need, risk, and operating cost. |
| PR.PT — Technology Infrastructure Resilience | Pushing reduction closer to the source improves pipeline efficiency and scalability. | |
| DE.CM — Continuous Monitoring | Derived metrics are part of monitoring design, but should not force unnecessary raw-data retention. | |
| Recommendation — Set observability retention and aggregation policies from business and risk requirements, not convenience. Design telemetry pipelines to reduce volume before central platform processing where practical. Balance monitoring coverage with the minimum raw telemetry needed to sustain detection and investigation. | ||
Practitioner Guidance
What to verify: Check whether each logged event is needed in full after the metric is produced, or whether the platform is retaining source data simply because it is convenient. If the answer is convenience, the design is usually over-paying for flexibility.
Decision rule: If the downstream use case only needs counters, rates, or coarse aggregates, move computation earlier in the pipeline and keep raw logs only where forensic or compliance value is clear. If teams still require broad search and replay, accept that platform-level metrics will remain a premium capability.
What practitioners underestimate: Observability cost is often driven less by dashboard volume than by the amount of raw telemetry that must remain queryable after metrics have already been derived. The cheapest signal is the one that is reduced before it becomes expensive to move, store, and scan.
Practitioner takeaway: Treat platform-level metric generation as a convenience feature with a storage tax, not as a free optimisation, and move reduction upstream whenever the raw log is no longer operationally essential.
Related resources from NHI Mgmt Group
- How should security and observability teams decide where to build log-based metrics in a telemetry pipeline?
- What is the difference between creating log-based metrics in an observability backend and building them in the telemetry pipeline?
- How should platform teams control Kubernetes observability costs without blocking developers?
- What breaks when an AI observability platform relies on a single warehouse or browser-based analysis layer?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 18, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
