Federated Security Search Guide: Querying Logs Where They Live Without Moving Everything to the SIEM

Security log volumes now grow faster than SIEM budgets, so many teams can no longer afford to ingest everything into one platform, yet investigators still need to search all of it. Federated security search (also called search in place or a decoupled SIEM) leaves data where it is stored, in object storage, data lakes, other SIEMs or regional clusters, and sends the query to the data instead. This guide explains the architectures, what they are good and bad at, how to decide which data stays in the SIEM, and how to roll out federation without losing detection coverage or evidence integrity. It is written for SOC architects, detection engineers and security data platform owners. Centralised lakes are covered in the Security Data Lake Guide; this page covers querying across stores. Products are named only as examples of a pattern and are listed alphabetically; order is never a ranking.

Key takeaways

• Federation is for investigation and hunting, not real-time detection: keep high-value, low-latency detection sources in the SIEM and federate the bulky, rarely queried data.
• Normalisation decides whether federation works: a query across stores is only useful if fields mean the same thing everywhere; a common schema such as OCSF makes that achievable.
• Open table formats change the economics: columnar files (Apache Parquet) and table formats (Apache Iceberg) let many engines query the same copy of data without re-ingesting it.
• Query cost and latency move to search time: scans of object storage are billed or slowed per query, so partitioning, time bounds and pre-built summaries matter.
• Access control must federate too: each store keeps its own permissions, so a federated layer needs service identities scoped per store and full audit of who searched what.

Why teams move to federated search

• Cost: high-volume sources such as flow logs, DNS query logs, proxy logs and cloud data events dominate ingestion bills but are queried mainly during investigations.
• Retention: regulations and investigation needs push retention to a year or more (see the Log Management and Retention Guide), which is cheaper in object storage than in hot SIEM indexes.
• Data residency: some data must stay in a country or region; federated queries can run where the data lives and return only results.
• Mergers and multiple SIEMs: acquired businesses and separate business units often run their own platforms; federation gives a single investigation view before (or instead of) consolidation.
• Data already exists elsewhere: cloud providers, data engineering teams and observability platforms already keep copies of security-relevant data.

Architectures compared

How the example building blocks behave

• Amazon Security Lake: AWS documents it as a managed data lake in the customer's own account, backed by Amazon S3, that converts natively supported AWS sources (CloudTrail management and data events, EKS audit logs, Route 53 Resolver query logs, Security Hub CSPM findings, VPC Flow Logs, WAFv2 logs) to OCSF and stores them partitioned in Apache Parquet. Subscribers get either notification-based access to new objects or query access through AWS Lake Formation tables, for example from Amazon Athena or Amazon Redshift, and can be limited to specific sources and Regions.
• Elastic cross-cluster search: Elastic documents API key authentication as the recommended security model for remote clusters, with TLS certificate authentication deprecated in 9.0. The skip_unavailable setting (default true) decides whether a failed remote is skipped or fails the search, and ccs_minimize_roundtrips controls how much work is pushed to remotes; ES|QL supports cross-cluster queries, subject to subscription level on both clusters.
• Kusto external tables: Microsoft documents an external table as a schema entity that references data stored outside the Kusto database, commonly CSV, JSON, Parquet or Avro files in Azure Blob Storage or Azure Data Lake, queried without ingestion.
• Trino: the Trino documentation describes connectors that adapt Trino to a data source, and catalogs that configure a connector for a specific source, so one SQL query can address several catalogs.

Availability notes: each vendor's documentation ties some capabilities to a subscription level, edition or Region; check the current documentation for the specific feature (for example, Elastic states ES|QL cross-cluster search requires the appropriate subscription on local and remote clusters).

Normalisation: the precondition

A federated query for "all logons by user X from IP Y" only works if every store calls those fields the same thing. Without a schema, analysts write a different query per store and miss data silently. The Open Cybersecurity Schema Framework (OCSF) is an open, vendor-neutral standard made up of categories, event classes, objects and an attribute dictionary, and is format agnostic (JSON, Parquet, Avro). Its categories include System Activity, Findings, Identity & Access Management, Network Activity, Discovery, Application Activity and Remediation. See the OCSF Guide.

Keep the raw event alongside the normalised one for evidence and for fields the schema does not cover. Version the schema in use, because OCSF itself evolves (the main development branch shows version 1.10.0-dev as of 8 November 2026).

Deciding what stays in the SIEM

Decision rule: if a source feeds a detection that must fire within minutes, keep at least the fields that detection needs in the SIEM. If a source is queried mainly after an alert to answer scope questions, federate it. Re-check placement every quarter against query logs: data searched daily belongs in a faster tier.

Small organisations: with modest volumes, a single SIEM with an archive tier is usually simpler and cheaper than running a lake and a query engine; federation adds operational work that needs data engineering skills. Mid-sized organisations: the SIEM-over-external-storage pattern often gives most of the saving with the least change for analysts. Large and multi-region organisations: a security data lake in an open format, with the SIEM holding detection data and cross-cluster or engine-based federation for regional stores, scales best, but only with a dedicated platform team that owns schema, partitioning and access control.

Performance and cost engineering

• Partition by time and source: almost every security query is time bounded; partition object storage by date (and often account or Region) so engines read only what they need.
• Use columnar formats: Parquet lets engines read only the referenced columns. Apache Iceberg adds table-level features its specification lists as goals: serialisable isolation, scan planning without listing every file, full schema and partition evolution, and planning reads from data value predicates rather than partition values.
• Compact small files: streaming pipelines create many small objects, which slows scans; schedule compaction.
• Pre-aggregate: daily summaries (unique domains per host, bytes per destination) answer most scoping questions without scanning raw events.
• Set query guardrails: enforce maximum time ranges and scanned bytes per query, and require a time filter.
• Measure cost per investigation: track scan charges by team so the cheaper storage does not turn into expensive queries.

Security of the federated layer: where identity fits

A federated search layer is a privileged non-human identity with read access to almost every log store, which makes it a prime target. Its credentials can reveal the whole security picture to an attacker and help them check whether they have been seen.

• Scope per store: use a separate service identity per data source with read-only access to the specific tables, buckets or indexes needed, preferring short-lived federated credentials and API keys with remote-specific privileges over static keys or shared certificates.
• Preserve least privilege for analysts: pass the analyst's identity or role through to each store where possible, rather than letting a broker query everything with one super account.
• Audit queries: log who ran which query, against which stores and with what result size. Unusual bulk exports are an insider and attacker signal.
• Protect log integrity: use write-once or object lock storage for evidence-grade logs and separate write and read permissions so the query layer cannot alter records.
• Data residency: confirm whether a federated query returns raw records across borders or only aggregated results.

See the Data Lake Security Guide and the Identity Telemetry Requirements.

Rolling out federated search

1. Inventory sources and queries: list every log source, its volume, retention, current location and which detections and saved searches use it. Pull query statistics from the SIEM to see what is actually searched.
2. Agree the schema: choose a common schema (for example OCSF) and map the top identity, network, endpoint and cloud fields first.
3. Pilot one bulky source: move one high-volume source (flow logs are a common first candidate) to the lake, keep it dual-written to the SIEM for 30 to 60 days, and compare investigation results and query times.
4. Test detection parity: replay known attack scenarios and confirm every detection still fires. Detections that need the data in real time stay on the SIEM copy or move to a streaming rule in the pipeline.
5. Cut over with rollback: stop SIEM ingestion only after sign-off, and keep the pipeline able to resume forwarding to the SIEM within hours if a gap appears.
6. Train and template: give analysts saved federated queries for common scoping questions and document latency expectations.
7. Expand and review: add sources in order of cost saving and low detection dependency; review placement quarterly.

Detection and measurement

• Detection parity: percentage of detection rules validated as firing after each source move (target 100 percent).
• Investigation query latency: median and 95th percentile time for standard scoping queries against federated data.
• Cost per GB retained and searched: total cost of storage plus scans compared with the previous SIEM ingestion cost.
• Federated query failure rate: share of queries returning partial results because a store was unavailable or timed out.
• Schema coverage: share of sources mapped to the common schema for core fields.
• Audit coverage: share of federated queries with complete user and store audit records.

Framework mapping

Common failure patterns

• Moving detection sources out of the SIEM: a cost project relocates a source that feeds real-time rules, and detections stop firing silently.
• No common schema: each store uses different field names, so federated queries return partial results that look complete.
• Silent partial results: a remote store is unavailable and the query engine skips it by design (for example when remotes are configured to be skipped on failure), so analysts conclude there was no activity.
• One over-privileged broker account: the federation service holds static, organisation-wide read keys that nobody rotates or monitors.
• Unbounded scans: open time ranges against raw object storage produce slow, expensive queries and analysts give up on the lake.
• Evidence without integrity: lake data is mutable by the same pipeline identity that writes it, weakening its value in forensics and legal proceedings.

Practitioner checklist

• Inventory sources with volume, retention, location and dependent detections, and pull SIEM query statistics.
• Choose a common schema, map core identity, network, endpoint and cloud fields, and keep raw events.
• Keep real-time detection fields in the SIEM; federate high-volume scoping data.
• Store lake data in columnar, time-partitioned formats with compaction and pre-built summaries.
• Show partial-result warnings to analysts whenever a federated store fails or is skipped.
• Give the federation layer per-store, read-only, short-lived credentials and audit every query.
• Use immutable storage for evidence-grade logs and separate write from read permissions.
• Pilot each source move with dual writing and a detection parity test before cutting over.
• Report detection parity, query latency, cost per GB and query failure rate quarterly.

Standards and references

• OCSF: Open Cybersecurity Schema Framework schema repository
• Apache Software Foundation: Apache Iceberg Table Spec
• NIST: SP 800-53 Rev. 5
• NIST: Cybersecurity Framework 2.0
• Amazon Web Services: What is Amazon Security Lake?
• Elastic: Cross-cluster search
• Microsoft Learn: External tables (Kusto)
• Trino: Trino concepts

Related NHI Mgmt Group resources: Security Data Lake Guide · SIEM Guide · OCSF Guide · Security Data Pipeline Guide · Detection Engineering Guide