TL;DR: Federated search changes the economics of security data by removing the need to build and store indexes for every retained source, according to Query, while indexed search still commits teams to pre-decided questions and ongoing storage costs. The practical shift is toward placing hot, cold and external data where each belongs, then querying across it without re-platforming.
At a glance
What this is: NHI Mgmt Group's analysis, drawing on Query's blog post: Federated search is better suited than index-first architectures for retaining security data across heterogeneous sources without committing every dataset to a permanent index.
Why it matters: This matters because security and IAM teams increasingly need to search data where it already lives, including identity, cloud, EDR and audit systems, without creating a new storage, cost and governance burden for each source.
Context
Federated search is a query pattern that asks the source system to execute as much of the request as possible instead of copying everything into a separate search store first. That matters in security operations because the architectural choice determines retention cost, latency, governance scope and which data sources stay searchable at all.
The core trade-off is between pre-indexed speed and on-demand breadth. Index-first designs are strong when hunters repeatedly ask the same kinds of questions, but they create a standing storage and pipeline obligation. In identity-heavy environments, that also affects how easily teams can query workforce identity systems, EDR consoles, ticketing platforms and cloud control planes from one place.
Key questions
Q: How do index-first search engines handle security logs?
A: An index-first search engine stores security logs in an index optimised for querying, so analysts can filter by fields such as host, user, event type and time without scanning raw files each time. The key controls are parsing, field extraction, retention policy, access control on the index, and tamper-resistant log storage for forensic integrity.
Q: When should security teams use a column store instead of a log index?
A: Security teams should use a column store when the main workload is analytical queries across many rows but only a few fields, such as aggregation, reporting, and long-range trend analysis. Use a log index when the priority is fast event retrieval, full-text search, and low-latency investigation across semi-structured security logs.
Technical breakdown
Why inverted indexes are fast but commit you to a data model
An inverted index maps terms to the documents that contain them, which makes rare-string lookup extremely fast. The price is that tokenization, field extraction and model assumptions happen before the query, so the system decides what future questions will be cheap. That works well for hunting known patterns, but it weakens flexibility when analysts need aggregates, cross-source joins or searches that fall outside the original schema. The result is not just performance tuning. It is a governance decision about which questions the data platform is designed to answer cheaply. See the Security Data Lake Guide.
Practical implication: Treat index-first search as a selective optimization, not a universal archive strategy, when defining security data architecture.
What federated query changes for security data access
Federated query keeps the data in its source system and delegates the work to a planner that decides where each predicate runs. That means the engine can query object stores, warehouses, SaaS audit logs and operational platforms through one language without building a separate index for every source. In practice, the architecture trades some uniform speed for broader reach and lower duplication. It also changes where control lives, because the source system retains more of the execution responsibility and storage ownership.
Practical implication: Use federated query when the value lies in breadth across many systems of record rather than in prebuilt lookup speed.
Why column stores and accelerated connectors address analytics differently
Column stores organise values by field rather than by document, which makes aggregation, filtering and numeric analysis much cheaper than re-extracting fields from an inverted index. They also preserve type information, so sums, percentiles and range queries work natively instead of being reconstructed from text search output. In security telemetry, that matters because many recurring questions are analytical rather than term-based. The architectural point is that a search stack can mix retrieval styles, but the storage choice should match the dominant workload rather than forcing every question through the same engine.
Practical implication: Place high-frequency analytical workloads on columnar storage instead of forcing every detection and hunt through a term index.
Threat narrative
Attacker objective: The objective is not exfiltration but operational blind spots created by incomplete search coverage and costly data duplication.
- Entry occurs when teams rely on a single indexing model that only captures the systems and fields the schema anticipated.
- Escalation follows when analysts discover that identity platforms, ticketing systems or SaaS audit logs were never made first-class search sources.
- Impact is architectural lock-in, where the organisation pays ongoing storage and pipeline costs while still lacking complete query coverage.
Breaches seen in the wild
- Microsoft SAS token exposure 2023: An over-permissive Azure SAS token in a Microsoft AI GitHub repo exposed 38TB, including workstation backups and Teams messages, for 3 years.
- Salesloft OAuth token breach: hackers stole OAuth tokens to access Salesforce data via Salesloft.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Federated search is a governance decision about where security truth lives. Once a platform copies every source into its own index, it also becomes the place where retention cost, schema assumptions and query shape are controlled. That can simplify hunting, but it centralises operational risk and creates a standing obligation to keep the index current. Practitioners should treat search architecture as a control plane decision, not just a performance choice.
Identity systems should not be treated as second-class sources. The strongest architectural argument in Query's analysis is that many of the most useful security records live in systems with APIs rather than in a log lake. Identity providers, EDR consoles and ticketing platforms often hold the context that turns an alert into an investigation. If those sources are only reachable through manual export, workforce identity systems and adjacent control planes remain analytically isolated.
Hybrid search beats architectural purity. Some workloads need an index, some need a column store, and some need neither. The mistake is to force a single storage pattern across all security questions and then call the result a platform strategy. Teams should decide which datasets are worth indexing, which should remain queryable in place, and which analytical paths justify a separate accelerated store.
Cost visibility is part of search governance. The article makes a useful point that the real question is not whether a system can search retained data, but who pays when it does. If every rare hunt creates a storage bill or scan fee, teams will ration visibility whether they mean to or not. Security architects should insist on cost models that match actual investigation patterns, not vendor convenience.
From our research library:
- 28.65 million new hardcoded secrets were detected in public GitHub commits in 2025 alone, a 34% year-over-year increase and the largest single-year jump ever recorded, according to the State of Secrets Sprawl 2026.
- Read next: Security Data Lake Guide
What this signals
Search architecture is now a governance surface. When security telemetry spans identity providers, EDR, SaaS audit logs and cloud control planes, the question is no longer whether data can be searched, but whether the organisation has chosen the right retrieval model for each source. That choice determines who can investigate, how fast they can move, and which data sets remain economically searchable over time.
Query locality will shape the next generation of security operations. The more teams rely on federated search and federated query, the more they will design around source systems that remain authoritative instead of copying everything into one central repository. That favours architectures that can balance retention, cost and investigation breadth without forcing every record into the same storage pattern.
For practitioners
- Define search tiers by workload shape Place rare-indicator hunts, scheduled detections and historical analytics on different retrieval paths instead of forcing one architecture to handle all three equally well.
- Keep identity and security systems queryable in place Preserve direct query access to identity providers, EDR consoles, ticketing platforms and SaaS audit logs so investigations do not depend on export pipelines.
- Separate index decisions from retention decisions Choose whether a dataset should be indexed based on query frequency and analytical shape, not simply because it must be retained for compliance.
- Track the full cost of search across retained data Model storage, scan and pipeline costs together so investigative breadth does not get constrained by hidden per-query or per-index charges.
Key takeaways
- Federated search changes security data architecture by reducing the need to prebuild and store indexes for every retained source.
- Index-first designs still excel at repeated lookups over known patterns, but they impose persistent cost and schema commitments.
- Security and IAM teams should decide where data lives and how it is queried based on workload shape, source diversity and governance burden.
Related NHI Mgmt Group resources: Security Data Lake Guide · Cloud Misconfiguration Guide · Workforce Identity Security Guide · Cloud Security Statistics Guide
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST CSF 2.0, CIS Controls v8, NIST SP 800-53 Rev 5 and CSA Cloud Controls Matrix set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | ID.AM-01 — Physical Devices and Systems Inventoried | Federated search depends on knowing which systems and sources exist across the security stack. |
| PR.AA-05 — Access Permissions, Entitlements and Authorizations | Search over identity, EDR and audit systems still depends on governed access to the underlying sources. | |
| Recommendation — Inventory security data sources before deciding which systems should be indexed, federated or left in place. Apply PR.AA-05 to govern who can query each source system and what data they can retrieve. | ||
| CIS Controls v8 | CIS-6 — Access Control Management | Federated access to many security systems requires consistent entitlement governance across sources. |
| Recommendation — Use CIS-6 to standardise access decisions across identity, cloud and telemetry systems. | ||
| NIST SP 800-53 Rev 5 | AU-2 — Event Logging | The topic centres on querying retained logs and audit records across many platforms. |
| AU-6 — Audit Record Review, Analysis, and Reporting | Federated search is useful only if teams can review and analyse audit records across sources. | |
| Recommendation — Use AU-2 to ensure the logs you need for federated investigation are actually captured at the source. Apply AU-6 to make cross-source audit analysis a routine capability, not an ad hoc export process. | ||
| CSA Cloud Controls Matrix | SEF — Security Incident Management, E-Discovery and Cloud Forensics | Cloud forensics and incident response depend on searchable telemetry across heterogeneous sources. |
| Recommendation — Use SEF to align cloud forensics workflows with federated access to logs and evidence. | ||
Key terms
- Federated Search: Federated search is a query method that looks across multiple data stores without first copying everything into one central repository. In identity security operations, it helps teams preserve context across live, cold, and distributed sources while reducing duplication and storage lock-in.
- Federated Query Execution: The process of running a single investigative hypothesis across multiple data sources at the same time. In security operations, this matters because attacker behaviour is often only visible when SIEM, EDR, identity, and threat intelligence data are correlated together.
- Inverted Index: An inverted index is a data structure that maps terms to the documents or records that contain them. Search engines and log platforms use it to retrieve matches quickly, and access control can still govern which indexed documents a user is allowed to see.
- Column Store: A column store is a database layout that keeps values for each field together across many records. This design helps queries read only the needed columns, which improves analytical performance and can compress data efficiently. It is most useful when the workload scans large datasets, filters by a few attributes, or builds reports and trends.
Deepen your knowledge
NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, IAM and workload identity through practitioner-focused learning. It is designed for security leaders who need to connect identity controls with the broader operational systems their programmes depend on.
Published by the NHIMG editorial team on November 9, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org