Federated search is the missing layer in telemetry pipelines

At a glance

What this is: NHI Mgmt Group's analysis, drawing on DataBahn's blog post: Federated search adds cross-destination query capability to telemetry pipelines, letting teams find data where it already lives instead of centralising it first.

Why it matters: It matters because IAM, SOC, and cloud teams often distribute logs for cost and retention reasons, and investigations fail when historical data becomes hard to retrieve.

Context

A telemetry pipeline is about movement and shape, not investigation. It decides how data is collected, normalised, enriched, reduced, and routed, but once that data lands in a SIEM, object storage, or a lake, the routing layer has done its job and visibility can still fragment.

That gap shows up when security teams need older history across more than one destination. In mixed environments, cost optimisation pushes hot data into a SIEM and colder data into cheaper storage, so the operational question becomes whether investigators can still search across all of it without reingesting it first.

Key questions

Q: What breaks when telemetry is routed out of the SIEM but not searchable later?

A: The control break is not storage itself, it is discoverability. Teams lose the ability to answer historical investigative questions without reingesting data, which adds delay, cost, and operational friction. The result is a retention strategy that looks economical on paper but becomes brittle the moment an incident or audit depends on older evidence.

Q: Why do telemetry pipelines need federated search instead of centralising all data?

A: Centralising all telemetry is often too costly and slow for modern retention needs. Federated search lets teams keep data in the cheapest appropriate store while preserving investigation reach across those stores. The governance test is whether routing choices remain reversible enough for hunting, forensics, and compliance queries.

Q: How do security teams know federated search is actually working?

A: It is working when analysts can query recent and archived data together, across multiple stores, without reingesting the archive first. The practical test is a real investigation workflow, not a feature checklist. If the search returns complete results across destinations and preserves entity context, the control is doing its job.

Q: Should organisations centralise all telemetry instead of using distributed storage?

A: Not usually. Centralising everything raises cost and often creates new retention problems without solving investigation at scale. A better model is to route data to the right tier, then make that data searchable across tiers so cost, retention, and investigative reach can coexist.

Technical breakdown

Why routing and search are different layers

A telemetry pipeline governs data flow. It chooses where each event goes based on value, cost, age, or source, and it may normalise or enrich data on the way, but it does not create a durable cross-store query path. Federated search sits above that layer and issues a query to each connected destination, then reconciles results into one view. That matters because the same dataset can be optimized for different storage tiers without losing investigative reach. The architecture only works if the search layer can understand destination-specific access, schema variation, and response timing without forcing all data back into one system.

Practical implication: Treat routing and search as separate design decisions, not one capability.

How federated query preserves low-cost retention

Federated search avoids the costly loop of reingesting archived data into a SIEM just to answer a question. Instead of copying data back into the hot platform, it queries the SIEM for recent events and cold storage for older records in parallel. That preserves the original retention choice, which is often why the data was moved in the first place. From a governance perspective, this separates retention policy from investigative access. From an operational perspective, it reduces the pressure to keep everything in expensive indexed storage purely for fear of losing searchability later.

Practical implication: Use federated query to keep retention economics intact while maintaining investigative access.

Why entity context matters in distributed investigations

A federated query is only useful if the investigator can connect records that refer to the same host, user, or workload across different stores. That is where a knowledge graph or shared identity layer adds value: it resolves aliases, related assets, and linked entities so the search is not just keyword matching across silos. In identity-heavy environments, this matters for accounts, workloads, and delegated access paths that may appear under different names in different systems. Without that context, federated search returns volume; with it, it returns usable investigative meaning.

Practical implication: Preserve entity resolution across stores so searches stay analytically useful.

Threat narrative

Attacker objective: The practical objective is to exploit the visibility gap created by fragmented telemetry so investigations are slower, more expensive, and less complete.

1. Entry occurs when investigative data is scattered across multiple destinations after normal pipeline routing, not when an attacker gains access.
2. Escalation happens when analysts must reingest archived data into a SIEM to answer a time-sensitive question, creating delay and extra cost.
3. Impact is reduced investigative reach, because older evidence can remain technically retained but operationally inaccessible when teams need it most.

Breaches seen in the wild

• reviewdog Action compromise 2025: A stolen maintainer token poisoned reviewdog/action-setup, leaking CI secrets including the tj-actions bot token used in the next attack.
• CI/CD pipeline exploitation case study: Credentials in an exposed .git/config let a researcher edit a Bitbucket pipeline so it planted their SSH key on the server. No victim was named.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Federated search is becoming a governance layer, not just a convenience feature. Once organisations deliberately route logs into different destinations for cost and retention reasons, the real control question becomes whether evidence is still discoverable on demand. That changes telemetry architecture from a storage conversation into an investigation-readiness conversation. Practitioners should treat searchability as part of security design, not an afterthought.

Telemetry fragmentation is now normal, so the search model has to assume distribution. Most environments will not converge cleanly back into one platform because migrations stall, acquisitions create new stores, and retention economics keep pushing data outward. The field implication is that centralisation is no longer the default answer to visibility gaps. The better question is whether the query layer can span the stores you already operate.

Data control plane architecture should separate routing value from forensic value. A pipeline can optimise ingest, but it cannot by itself preserve investigability across Splunk, object storage, and lakes. That distinction is especially relevant where identity, workload, and cloud logs must be correlated across systems. Security leaders should design for retained reach rather than retained duplication.

Federated search reduces SIEM lock-in pressure without lowering governance standards. Teams can push more low-value telemetry into cheaper storage when they know they can still search it later, which is the real economic unlock. The governance risk is not lower visibility from using cheaper tiers; it is assuming a routing engine can substitute for a search layer. Practitioners should re-evaluate retention design, not just ingestion cost.

Cross-store search becomes more valuable when identity context is preserved. Telemetry is useful only when analysts can tie one host, account, or workload to the same entity across destinations. That means federated search should sit alongside entity resolution, not replace it. In practice, the most effective programmes will align storage tiering, search, and identity context as one operating model.

What this signals

Searchability is the missing governance condition in many telemetry programmes. Once teams accept that data will live in multiple destinations, the architecture has to preserve investigation rights across those stores instead of assuming a single index will hold everything. For SOC and cloud teams, that means checking whether routing policies and query paths are designed together rather than separately.

Federated search changes the economics of retention. When older logs remain searchable in cold storage, teams can reserve premium SIEM capacity for active investigation data without losing historical reach. The operational signal is simple: if your analysts can prove they can find six months of history without reingesting it, your retention model is mature enough to be aggressive.

Cross-store visibility becomes more reliable when identity context travels with the event. A search layer that can connect a host, account, or workload across systems gives investigators a coherent picture even when telemetry is split across tools. That is especially relevant where identity, workload, and cloud logs are part of the same incident trail.

For practitioners

• Separate routing policy from search policy Document which data is sent to Splunk, which data is archived, and which queries must still span both so the pipeline does not become a visibility boundary.
• Test cross-store investigations before forcing centralisation Run a real six-month-history query across hot and cold stores to confirm analysts can find evidence without reingesting archived data.
• Preserve entity resolution across telemetry stores Keep host, identity, and workload mapping consistent so federated queries can connect aliases and related records across different destinations.
• Use cheaper storage for low-value data with searchable retention Move routine or long-tail logs out of premium SIEM ingestion when the search layer can still retrieve them later for audit or incident response.

Key takeaways

• Telemetry routing and telemetry search solve different problems, and confusing them creates visibility gaps.
• Federated search lets teams keep cheaper retention tiers while still investigating older events across multiple stores.
• The strongest operating model is one that pairs distribution with searchable retention and entity context.

Key terms

• Federated Search: Federated search is a query method that looks across multiple data stores without first copying everything into one central repository. In identity security operations, it helps teams preserve context across live, cold, and distributed sources while reducing duplication and storage lock-in.
• Telemetry pipeline: A telemetry pipeline is the path security data follows from collection to analysis and retention. In mature environments it must preserve context, maintain throughput, and avoid introducing blind spots as sources, formats, and volumes change over time.
• Cross-Store Investigability: The ability to find, correlate, and analyse evidence across more than one storage system without first consolidating it. It is a governance outcome as much as a technical one, because it determines whether retention and cost optimisation weaken incident response.
• Entity Resolution: Entity resolution is the process of determining which accounts, wallets, devices, or records belong to the same real-world actor. It is essential for sanctions and AML programmes because transaction data alone often hides related activity unless it is linked back to a trusted identity model.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management in a way that supports broader security operations. It is suited to practitioners who need to connect identity control to telemetry, investigation, and resilience.