Should organisations centralise all telemetry instead of using distributed storage?

Why centralising every telemetry stream usually backfires

Telemetry architecture is a storage and retrieval problem, not a single-repository problem. Once all events, logs, and traces are forced into one place, the organisation inherits one expensive retention model for everything, even though different data sets have different value, volume, and compliance pressure. That usually increases spend, slows ingestion choices, and creates harder retention decisions without improving investigation quality.

The better design is to separate collection from analysis. Keep hot, warm, and cold tiers for the right data classes, then make those tiers searchable from one investigative workflow. That preserves operational flexibility while still letting analysts correlate across sources when they need to rebuild an incident timeline.

A distributed model also reduces the temptation to treat all telemetry as equally important. Some data needs rapid query performance, some needs cheap long-term retention, and some can be summarised or sampled once it has served its immediate purpose. The architectural mistake is assuming centralisation automatically equals visibility; in practice, searchability and good indexing matter more than physical co-location.

How tiered search supports investigation at scale

The practical question is whether the organisation can answer investigative queries quickly enough across heterogeneous storage. If the platform can search across tiers, centralisation becomes less important than metadata quality, schema consistency, and query federation. Analysts care about being able to pivot from a high-signal alert to the surrounding context, regardless of which tier or system retained that context.

This is where routing policy matters. High-value telemetry should land where it can be queried cheaply enough to support routine detection, while lower-frequency or compliance-oriented data can stay in lower-cost tiers. A search layer that understands those placements lets teams retain evidence longer without paying premium storage costs for every byte.

Distributed storage also helps when teams have different operational needs. Security operations, platform engineering, and compliance often do not need the same retention window or retrieval speed. By keeping the underlying storage distributed but presenting a unified search path, organisations can align performance, cost, and evidence retention to the actual use case instead of forcing one compromise for all of them.

When centralisation is still useful, and where it is not

Centralisation can still be useful for narrow, high-value telemetry classes, especially where parsing, enrichment, or real-time correlation depend on a common processing layer. It is also helpful when the same data feeds multiple detection or reporting workflows and the volume is manageable. The issue is not central storage itself, but assuming every source deserves the same treatment.

For high-volume platforms, centralising raw data often creates a bottleneck before it creates insight. A more resilient pattern is to centralise governance and query control, not necessarily every raw event. That way, the organisation can standardise access, retention policy, and evidence handling without collapsing all ingestion into one storage engine.

Good telemetry design therefore distinguishes three things: where data is kept, how long it is retained, and how easily it can be searched. Those are related but not identical decisions, and confusing them is what produces either runaway storage costs or brittle investigations.

Risk and Threat Considerations

Centralising all telemetry concentrates cost, retention, and availability risk in one control plane. It can also create a single operational choke point if ingestion, indexing, or access control fails, while distributed storage introduces risk when teams cannot search across tiers consistently enough to investigate incidents.

Failure mechanism: A one-size-fits-all storage model forces expensive retention for low-value data, or premature deletion for high-value data, and either choice weakens the organisation’s ability to investigate security events.

Impact: Teams lose either cost discipline or investigative reach, and in the worst case they lose both, because the telemetry is present but not practically usable when an incident occurs.

Practitioner Guidance

What to prioritise: Design the query layer before standardising the storage layer. If investigators cannot search across tiers with acceptable latency and usable metadata, centralisation will not solve the real problem.

Decision rule: Keep data distributed when retention windows, access frequency, and cost profiles differ materially; centralise only the governance, routing policy, and search experience. Treat raw centralisation as a storage decision, not an observability strategy.

What to verify: Confirm that the platform can retrieve a complete incident timeline from hot and cold tiers without manual data copying. If the answer depends on ad hoc exports, the design is already too fragmented for reliable response.

Practitioner takeaway: The best telemetry architecture is the one that preserves evidence, controls cost, and keeps investigation fast. That usually means distributed storage with unified search, not a single warehouse for everything.

Related resources from NHI Mgmt Group

• When should organisations use AI for vendor risk management instead of only using it for document storage?
• Should organisations centralise NHI governance instead of using separate scanners and vaults?
• When should organisations centralise privilege policy instead of using system-specific controls?
• Should organisations centralise secret storage or standardise secret governance first?