Security Data Lake Guide: Architecture, OCSF Mapping and Log Analytics

A security data lake is meant to give the SOC years of searchable telemetry at storage prices, but many become a data swamp: data nobody can find, query in time or trust. The difference is design: an agreed schema, a catalogue, a tiering plan, an access model and detections that actually run against the lake. This guide goes below the pipeline into the lake itself: architecture, OCSF field mapping and extensions, schema drift, hot and cold log storage, querying and federated search, and securing the lake. For collection, parsing and routing upstream, see the Security Data Pipeline Guide. It is for detection engineers, security data architects and SOC leads; examples use Amazon Security Lake, Azure Data Lake Storage, Log Analytics workspaces and Google Cloud.

Key takeaways

• A lake without detections is an archive. Decide which hunts, long-window detections and investigations will run against it before choosing storage, formats or tiers.
• Map to OCSF once, keep the original. One event class per source, required attributes populated, the rest in unmapped and the source record kept, so mapping mistakes can be replayed.
• Schema drift is the quiet failure. A renamed vendor field or new schema version breaks saved queries without an error. Pin versions, test mappings and use a table format that tracks columns by ID.
• Tier by query need. Hot storage for real-time detection, a queryable lake tier for hunting, a cold storage tier for retention, and a measured time to bring cold data back.
• The lake is a crown-jewel store. Give it its own account, narrow non-human identities, per-source access and an immutable copy its own administrators cannot delete.

Data lake vs data warehouse vs SIEM

A data lake for security stores telemetry in open, columnar formats on object storage, separates storage from compute and lets several tools query the same data. Amazon Security Lake describes itself as "a fully managed security data lake service" backed by S3 buckets where "you retain ownership over your data". Ownership is the main difference between security data lakes and a SIEM, where data usually lives inside the product.

Schema on read vs schema on write: a pure schema-on-read lake makes every analyst rediscover field meanings, which is how data swamps form. The working compromise is a normalised OCSF layer written at ingest plus raw objects for reinterpretation; the curated layer on top is often called the security data warehouse.

Unified or fragmented: a unified data lake for security, IT and identity telemetry reduces copies if access control is fine-grained. A fragmented data lake estate (one per cloud, business unit or tool) creates blind spots in any investigation that crosses them. Where residency forces fragmentation, catalogue every lake centrally and plan federated search.

Reference architecture

Amazon Security Lake

• OCSF native ingestion: it "converts ingested data into Apache Parquet format" and transforms natively supported AWS sources to OCSF: CloudTrail management and data events, EKS audit logs, Route 53 resolver query logs, Security Hub CSPM findings, VPC Flow Logs and WAFv2 logs.
• Catalogue and subscribers: one Lake Formation table per source; subscribers are notified of new objects or query the lake, and see only the sources and Regions you grant. Rollup Regions consolidate data, so check residency first.
• Limits: up to 50 custom sources per account, and "Security Lake doesn't support Amazon S3 Object Lock", so an immutable legal-hold copy must live elsewhere.

Azure Data Lake Storage and Log Analytics workspaces

Azure Data Lake Storage is Blob Storage with the hierarchical namespace enabled, so directory rename and delete are "single atomic metadata operations", with Azure RBAC and POSIX ACLs per directory or file and the ABFS endpoint (dfs.core.windows.net) for Spark and Presto. It suits a self-built lake. A Log Analytics workspace is "a data store that holds tables", queried with KQL, with a plan per table (see tiering). Many organisations use the workspace for detection and the storage account for raw archives.

Google Cloud

Upgrading a Cloud Logging log bucket enables SQL analytics, and a linked BigQuery dataset lets you "query your log data from BigQuery, without exporting that data". Backfill "might take several days", and BigQuery does not honour Cloud Logging field-level access controls, so dataset readers see every field.

OCSF mapping in the lake

The parent pipeline guide explains the schema; this section covers OCSF mapping decisions that matter once data is stored for years.

• One class per stream: OCSF intends "the mapping of any raw event to a single event class". For custom sources Security Lake requires that "The same OCSF event class should apply to each record within a Parquet-formatted object", with each class delivered as a separate source.
• Honest enums: Unknown (0) when data is missing; Other (99) with the sibling string when no value fits. Never guess an activity_id to pass a validator.
• Unmapped fields are data: fields with no schema target travel in the optional unmapped attribute. Keep them; track the share of events carrying them as a mapping-quality signal.
• Keep the original: raw_data "holds the event data as received from the source"; for large events store raw objects separately and keep a pointer.
• Record the version: metadata.version is required. For custom sources "Security Lake supports OCSF version 1.3 and earlier", even though the schema has moved on.
• Populate observables: the observables array lets analysts search for an IP or hash without knowing which class holds it, which makes cross-source data lake search practical.

Example: OCSF field mapping for an identity sign-in

Sign-ins are the most valuable stream in an identity data lake (an identity security data lake holds IdP, directory, cloud IAM and PAM events together). In the current schema Authentication is class 2 in Identity & Access Management (category_uid 3), so class_uid 3002, and a logon gives type_uid 300201 (class_uid * 100 + activity_id). Some older OCSF examples still show 3001: exactly the drift to test for.

OCSF extension: extensions add attributes, objects, profiles or classes "without modification of the core schema", live in their own folder with dictionary.json, events and objects, and register a UID in the OCSF Extensions Registry so class IDs do not collide. Write one when an unmapped field is used by detections across several sources; until then, unmapped is cheaper.

Schema drift and data quality

Schema drift is any change in the shape or meaning of incoming data: a renamed field, a new nested object, a changed timestamp format, a new OCSF version. In a lake it rarely fails loudly: queries return fewer rows and detections go quiet.

• Track columns by ID: Apache Iceberg "uses unique IDs to track each column", schema updates are "metadata changes, so no data files need to be rewritten", and "Added columns never read existing values from another column". Name-tracked formats can "inadvertently un-delete a column if a name is reused". Partition evolution is metadata-only too.
• Contract tests: sample events per source and version in CI; fail the build if required attributes go empty or types change.
• Drift monitors: alert on new top-level fields, rising unmapped population, null rates on required attributes and new type_uid values.
• Time discipline: partition and query in UTC. Security Lake's eventDay partition is UTC, and the 2024 joint event logging guidance led by ASD's ACSC recommends UTC and ISO 8601 with millisecond precision.

Data tiering: hot and cold log storage

The joint guidance describes hot storage for logs that must be readily searchable and cold storage for lower-priority, cheaper archives. Each data lake tier gives something up.

• Set retention through the lake: AWS warns that editing S3 Lifecycle rules directly "can potentially delete metadata and prevent you from accessing your data". Without settings, data stays in S3 Standard indefinitely.
• Measure the restore path: time to bring a month of cold data back into a queryable state, written into the incident response plan. See the Log Management and Retention Guide for retention periods.

Querying, detection and federated search

Data lake analytics suits what a SIEM handles badly: long windows, high volume and rare events.

• Long-window detections: beaconing across weeks of DNS or flow data, a service account used for the first time in 90 days, a token used from a new network after months of stable use.
• Retroactive hunts: search a year of telemetry when a new technique or indicator is published.
• Scoping: enumerate every action by a compromised identity across all sources.

Threat-informed log analytics: the joint guidance names a "detection strategy for relevant threats" as one of four key factors, with an enterprise-approved logging policy, centralised access and correlation, and secure storage. Map the ATT&CK techniques in your threat model to the sources and OCSF classes that reveal them, onboard those first, and track coverage. See the Detection Engineering Guide.

Data lake detection mechanics: scheduled queries write results to a findings table or raise SIEM alerts, rather than promoting raw data to the hot tier. Log Analytics summary rules aggregate data "as the data arrives" (single table only on Basic and Auxiliary), turning verbose logs into compact detection inputs.

Federated search: querying data where it lives. A linked BigQuery dataset reads log buckets without ingesting; Security Lake query subscribers read Lake Formation tables over data that stays in your S3 buckets. Each federated query inherits the slowest store and weakest access control, so rehearse it before an incident.

Cost control: partition by time, source and account, order records by time inside files (Security Lake asks this "to reduce the cost of querying data"), and give analysts saved hunts with a required time range.

Securing the lake and its identities

The lake concentrates the evidence an intruder wants to destroy and the personal data regulators care about. Attackers collect from cloud storage (ATT&CK T1530 Data from Cloud Storage) and switch off logging (T1685.002 Disable or Modify Cloud Log, whose detection analytics include unexpected StopLogging or DeleteTrail calls and stopped audit exports). The joint guidance asks for protection "from unauthorised access, modification and deletion".

• Isolate: a dedicated log archive account or subscription, administered separately from production and from day-to-day lake operators.
• Narrow non-human identities: crawlers, ETL functions, source writers (Security Lake names them AmazonSecurityLake-Provider-{source}-{region}) and subscriber roles. Writers write to their own prefix with no delete; subscribers read named sources only; federation, not static keys. See the Service Account Security Guide.
• Column-level controls: restrict command lines, URLs with query strings and message bodies, which carry secrets and personal data.
• Keys: customer-managed keys with administrators separate from readers; alert on key policy changes, because deleting a key deletes the data.
• No long-lived shareable links: signed URLs and shared access tokens bypass every role. The Microsoft SAS token exposure showed one over-permissive link exposing 38TB of internal data.
• Watch the lake: alert on subscriber creation, permission grants, retention changes, bulk reads by unusual principals and source silence, to a destination lake administrators do not control.

Data lake onboarding, step by step

1. Define use cases: the hunts, long-window detections and investigation queries the lake must support, with sources and an owner for each.
2. Fix the contract: OCSF version, extension policy, partition keys, table format and raw retention location, written as the onboarding standard.
3. Build the landing zone: dedicated account, keys, catalogue, per-Region retention and immutable archive. Test that writers cannot delete and subscribers cannot read outside their grant.
4. Pilot three sources: identity provider sign-ins, cloud control plane audit and DNS or flow logs. Validate mappings, compare record counts with the source for two weeks and run the planned queries.
5. Onboard by wave in threat-model priority, each source with mapping tests, a drift monitor and a named consumer. Do not onboard a source nobody will query.
6. Move detections in parallel: run lake detections beside existing SIEM versions for a full cycle, compare, then retire duplicates.
7. Roll back by replay: version mapping code, keep raw objects so a bad mapping can be reprocessed, and never let a lake change delete the only copy.

Decision guidance

• Managed or self-built: a managed lake gives OCSF native ingestion, a catalogue and subscriber management with its own limits (source counts, OCSF versions, no object lock). A self-built lake on Azure Data Lake Storage, S3 or Cloud Storage with Iceberg tables gives full control and full engineering burden.
• Beside the SIEM, not instead: real-time correlation and cases stay in the SIEM; volume, history and hunting move to the lake. See the SIEM Guide.
• Small organisations: use the workspace or managed lake native to your main cloud, low-cost plans for verbose logs and long-term retention for compliance; do not build a lake you cannot staff.
• Large or multi-cloud organisations: a platform team running the lake as a product, OCSF as the internal contract, Iceberg tables, a central catalogue and federated search where data must stay in Region.

Detection and measurement

• Source freshness: time since the last record against expected cadence; silence is a data lake blind spot until proven otherwise.
• Mapping quality: share of records with all required OCSF attributes, and share carrying unmapped fields.
• Use: sources with at least one detection or saved hunt, and sources queried in the last 90 days.
• Coverage: threat-model ATT&CK techniques with a SIEM or lake detection.
• Time to answer: runtime of a standard 12-month scoping query, and restore time for a month of cold data.
• Access hygiene: lake roles reviewed this quarter; static keys with lake access (target zero).

Framework mapping

Common failure patterns

• The data swamp: everything onboarded, nothing catalogued, no owners, so analysts return to the SIEM.
• Silent schema drift: a field rename or OCSF version change empties a column and long-window detections stop firing.
• Retention changed underneath the service: direct lifecycle edits delete table metadata or evidence.
• The lake as the only copy: no immutable archive, so stolen keys can delete or encrypt the evidence, as in the Codefinger S3 ransomware.
• Over-broad subscribers: analytics teams and vendors read every source, including sign-ins and command lines full of personal data and secrets.

Practitioner checklist

• List the hunts and detections the lake must support before onboarding any source.
• Map each stream to one OCSF class, record metadata.version, keep unmapped fields and the raw record.
• Test mappings in CI and alert on schema drift.
• Use a table format with column IDs, such as Apache Iceberg, for self-built lakes.
• Set hot, lake and cold tiers per data class, manage retention through the lake service and measure restore time.
• Keep an immutable archive in a separate account that lake administrators cannot delete.
• Scope every writer, crawler and subscriber to named sources and Regions, with no delete and no static keys.
• Alert on source silence, permission and retention changes, and attempts to stop cloud logging.

Standards and references

• OCSF: Understanding OCSF
• OCSF: Open Cybersecurity Schema Framework schema repository
• ASD's ACSC, CISA and partners: Best practices for event logging and threat detection
• NIST: SP 800-92 Rev. 1 Cybersecurity Log Management Planning Guide (initial public draft)
• AWS: Collecting data from custom sources in Security Lake
• Microsoft: Azure Monitor Logs overview
• Apache Iceberg: Evolution
• Google Cloud: Log Analytics in Cloud Logging

Related NHI Mgmt Group resources: Security Data Pipeline Guide · SIEM Guide · Log Management and Retention Guide · Cloud Logging and Audit Trail Guide · Detection Engineering Guide