TL;DR: Telemetry pipelines decide where data is processed and routed, while federated search solves the separate problem of finding it again across Splunk, S3, and other destinations without reingesting everything first, according to DataBahn. That split lets teams optimize SIEM cost and retention without losing investigative reach, which becomes critical when history lives in multiple places.
At a glance
What this is: This is an analysis of why telemetry pipelines and federated search solve different problems, and why search across distributed data becomes essential once logs are routed into multiple destinations.
Why it matters: It matters because security and identity practitioners often treat routing and retrievability as one design choice, when in practice cost optimisation, retention, and investigation readiness depend on separating them.
👉 Read DataBahn's analysis of federated search and telemetry routing
Context
Telemetry architecture often fails when teams assume the system that routes data can also reliably retrieve it later. In practice, the pipeline decides where logs, events, and records go, but it does not guarantee that six months of history will remain cheaply searchable across every destination.
That distinction has a governance dimension for identity and access investigations as well, because the same data fragmentation that affects SOC work also affects auditability, forensic reconstruction, and later review of privileged or non-human activity. As data spreads across SIEM, data lake, and object storage, teams need searchability that survives routing decisions.
This is a classic distributed-data problem rather than a product-selection problem, and it is typical in mature environments that have optimised for cost, retention, and operational flexibility at the same time.
Key questions
Q: How should security teams design telemetry storage when data lives in multiple places?
A: Design the storage model around two separate questions: where data should live for cost and retention, and how it will be found later for investigation. Use the pipeline to route telemetry intelligently, but require a search layer that can query SIEM, data lake, and archive storage together. Without that separation, cost savings often come at the expense of investigative reach.
Q: Why does federated search matter when teams already have a SIEM?
A: A SIEM is a high-value analysis and retention platform, not a universal repository for every log. When teams route older or low-value data elsewhere, federated search preserves the ability to query that data without reingesting it. That matters for long lookbacks, audits, and incident reconstruction, especially when the evidence trail spans several storage tiers.
Q: What breaks when telemetry routing and search are treated as the same problem?
A: Teams eventually lose visibility into data that has been moved for cost or retention reasons. The routing layer can send records to cheaper storage, but it cannot make them searchable across every destination later. When an audit or incident demands history, analysts face delays, duplicate ingestion costs, and incomplete context.
Q: How can organisations know if their cross-store search is actually working?
A: Run realistic lookback tests that span recent SIEM data, colder object storage, and any data lake or archive you rely on. If analysts can rebuild the timeline, resolve the right entities, and complete the query without reingesting data, the design is working. If not, the search layer is not compensating for fragmentation.
Technical breakdown
What a telemetry pipeline actually controls
A telemetry pipeline is the control plane for collection, normalization, enrichment, reduction, and routing. It decides which records go to a SIEM, which are archived in cheaper storage, and which are dropped because they add little investigative value. That makes it a cost and data-shaping mechanism, not a search mechanism. Once data lands in Splunk, S3, or a data lake, the pipeline has already completed its job. The architectural mistake is expecting a routing layer to also function as a distributed retrieval layer.
Practical implication: Separate routing design from retrieval design so storage decisions do not silently break later investigations.
How federated search differs from centralised ingestion
Federated search queries each destination where data already lives instead of moving all data into one platform first. That means recent records can remain in a SIEM while older or lower-value data stays in cold storage, yet both remain searchable through one interface. This matters because many organisations fragment data naturally through migrations, acquisitions, cloud logging, and retention policy changes. Federated search is the mechanism that preserves visibility after those decisions are made, rather than forcing a costly reingestion cycle every time investigators need history.
Practical implication: Use federated search when retention and cost require multiple repositories, but investigative access still has to be immediate.
Why entity context matters in cross-store investigations
Search becomes more useful when it understands relationships, not just keywords. A living knowledge graph can tie hostnames, identities, and assets together so investigators do not have to manually correlate equivalent records across systems. That is especially relevant when identity-related activity spans multiple stores, because the same service, user, or workload may appear differently in each source. For practitioners, the lesson is that cross-store search without entity resolution still leaves too much manual correlation work on the analyst.
Practical implication: Add entity and identity resolution if your search layer must support investigations across fragmented telemetry.
NHI Mgmt Group analysis
Federated search is not an enhancement to telemetry routing. It is the compensating control for a distributed retention strategy. Once teams route data to lower-cost storage, they have already accepted that the SIEM will not hold everything. The governance question then becomes whether that data remains findable when an investigation needs it. That is a data-access problem with identity and audit consequences, not just a storage problem. Practitioners should treat searchability as part of the control design, not as a convenience feature.
Telemetry fragmentation is now a normal state, not an exception. Migrations, acquisitions, cloud services, and compliance retention all push data into multiple repositories over time. The named concept here is searchability debt: the operational and investigative cost created when organisations optimise storage before they preserve cross-store retrieval. The debt grows quietly until an incident, audit, or access review forces a full-history query. Teams should assume fragmentation and design for it upfront.
Identity investigations are one of the clearest examples of why federated search matters. Privileged activity, service accounts, and workload events often span SIEM, object storage, and cloud-native logs. If those records are not searchable together, analysts lose the ability to reconstruct sequence and scope with confidence. That is especially problematic in environments with both human and non-human identities, where the evidence trail is already distributed. Practitioners should make entity-aware retrieval part of their identity evidence model.
Cost optimisation only works when retrieval is decoupled from ingestion. The old assumption was that if data was not in the expensive platform, it was effectively unavailable. That assumption no longer holds if federated search is in place. As a result, teams can be more aggressive about routing low-value logs away from premium storage without sacrificing later investigation capability. Practitioners should use that flexibility to redesign hot and cold data placement deliberately, not by habit.
What this signals
Searchability debt: once telemetry is split across SIEM, data lake, and cold storage, the real control objective becomes cross-store retrievability, not centralisation. Teams that over-optimise for ingest cost without preserving query access eventually pay back that debt during audits, incident review, and identity investigations. The practical signal is simple: if an analyst cannot run one lookback across hot and cold data, the architecture is already underpowered for the questions the business will ask.
In identity-heavy environments, federated search supports evidence continuity across human and non-human activity, which is increasingly important as service accounts, workload logs, and privileged actions land in different systems. That aligns with the searchability and access-governance expectations in NIST SP 800-53 Rev 5 Security and Privacy Controls, especially where access and auditability need to survive data movement. The programme-level implication is that retrieval design should sit beside storage design in every logging architecture review.
For practitioners
- Map routing and retrieval as separate controls Document which telemetry destinations are used for hot investigation, compliance retention, and archive storage, then verify that each is still searchable through a single query path.
- Test six-month lookbacks across every store Run an investigation scenario that spans SIEM, object storage, and data lake history so you can measure whether analysts can reconstruct an incident without reingesting data.
- Preserve entity relationships in the search layer Make sure identities, hosts, and workloads resolve to consistent entities across stores so analysts can correlate related records without manual field-by-field stitching.
- Use routing to reduce SIEM pressure deliberately Move low-value or infrequently queried telemetry to cheaper storage only after confirming that federated search can still retrieve it when needed.
Key takeaways
- Telemetry pipelines and federated search solve different problems, and conflating them creates blind spots.
- Distributed storage only works operationally when investigators can query all relevant history without reingesting it first.
- Teams that optimise retention and cost should treat cross-store searchability as a control objective, not a convenience feature.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK address the attack and risk surface, while NIST CSF 2.0, NIST SP 800-53 Rev 5 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.AC-4 — Access Permissions and Authorisations | Search across distributed telemetry depends on governed access to data stores and retrieval paths. |
| Recommendation — Map cross-store query access to PR.AC-4 and confirm analysts can retrieve needed history without reingesting data. | ||
| NIST SP 800-53 Rev 5 | AU-6 — Audit Record Review, Analysis, and Reporting | Federated search exists to support later review and analysis of retained records across stores. |
| Recommendation — Apply AU-6 to ensure retained telemetry remains reviewable across SIEM, data lake, and archive tiers. | ||
| CIS Controls v8 | CIS-8 — Audit Log Management | The article centers on preserving searchable telemetry after routing decisions are made. |
| Recommendation — Use CIS Control 8 to retain and search audit logs across all storage destinations used by the programme. | ||
| MITRE ATT&CK | TA0007 — Discovery | Investigations depend on discovery of records and relationships across fragmented telemetry stores. |
| Recommendation — Correlate discovery use cases to TA0007 and validate that analysts can locate relevant historical evidence quickly. | ||
Key terms
- Telemetry pipeline: A telemetry pipeline is the path security data follows from collection to analysis and retention. In mature environments it must preserve context, maintain throughput, and avoid introducing blind spots as sources, formats, and volumes change over time.
- Federated Search: Federated search is a query method that looks across multiple data stores without first copying everything into one central repository. In identity security operations, it helps teams preserve context across live, cold, and distributed sources while reducing duplication and storage lock-in.
- Searchability Debt: Searchability debt is the operational cost created when organisations optimise storage, retention, or routing before preserving reliable retrieval across all destinations. It usually surfaces later during audits or incidents, when teams discover that data exists but is slow or difficult to find.
- Entity Resolution: Entity resolution is the process of determining which accounts, wallets, devices, or records belong to the same real-world actor. It is essential for sanctions and AML programmes because transaction data alone often hides related activity unless it is linked back to a trusted identity model.
What's in the full article
DataBahn's full article covers the operational detail this post intentionally leaves for the source:
- How Smart Edge, Highway, and Cruz divide telemetry collection, transformation, and routing responsibilities
- How federated search runs across Splunk, S3, and data lake destinations without reingestion
- How the Reef knowledge graph supports entity resolution during investigations
- How MCP Hub extends search and orchestration to AI models and agents
Deepen your knowledge
NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps identity and security practitioners connect telemetry, access, and lifecycle controls to the evidence their programmes depend on.
Published by the NHIMG editorial team on September 11, 2026.
NHI Mgmt Group — the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
