Notifications
Clear all
Topic starter
18/08/2026 2:19 pm
TL;DR: OCSF standardises security event JSON, but the storage layout still depends on the engine: Iceberg can tolerate a wide, sparse table because width is mostly metadata cost, while ClickHouse usually needs promoted columns and payload spillover to stay performant, according to TENZIR. The practical takeaway is that schema design for OCSF is an engine decision, not a one-size-fits-all pattern.
NHIMG editorial — based on content published by TENZIR: OCSF storage strategies for Iceberg lakehouses and ClickHouse-style OLAP engines
By the numbers:
- Altinity measured 64 GB of peak RAM and 167 seconds to insert one million rows into a 10,000-column ClickHouse table.
- 6.5 GB of peak RAM and 12 seconds, RAM and 12 seconds at 1,000 columns, showing how width changes ClickHouse cost curves.
Questions worth separating out
Q: How should security teams choose an OCSF storage layout for different engines?
A: Start with the engine, not the schema.
Q: Why do wide OCSF tables work better in Iceberg than in ClickHouse?
A: Iceberg stores sparse columns cheaply because absent values cost nothing in a file and schema changes are mostly metadata updates.
Q: What breaks when OCSF is modelled as one giant table in the wrong engine?
A: The main failure mode is not schema correctness but operational cost.
Practitioner guidance
- Match table strategy to the storage engine Use a wide nested table in Iceberg when schema evolution and sparse columns are the norm, but prefer promoted columns with payload spillover in ClickHouse-style engines.
- Cap metadata growth before it affects planning Track manifest size, column statistics coverage, and partition-spec changes so width does not degrade Iceberg query planning.
- Promote only hot fields in OLAP pipelines Keep high-frequency attributes as typed columns and leave rare attributes in a semi-structured payload so insert and merge costs stay controlled.
What's in the full article
TENZIR's full article covers the operational detail this post intentionally leaves for the source:
- The exact OCSF table strategies compared across Iceberg and ClickHouse implementations
- The to_iceberg operator behavior for table creation, partitioning, and schema evolution
- The practical mechanics of Parquet file sizing, stats collection, and hidden partitioning
- The ClickHouse promotion pattern used by production SIEM-style deployments
👉 Read TENZIR's analysis of OCSF storage strategies in Iceberg and ClickHouse →
OCSF in Iceberg or ClickHouse: which storage layout holds up?
Explore further
18/08/2026 3:11 pm
Engine-aware OCSF design is now a governance choice, not a storage detail. OCSF gives teams a shared event vocabulary, but it does not tell them how to organise data for analysis. That omission matters because the wrong table strategy can create metadata drag, query latency, or operational complexity that undermines telemetry value. Practitioners should treat storage layout as part of security data governance, not a downstream implementation detail.
A question worth separating out:
Q: How can teams decide whether to use promoted columns or semi-structured payloads?
A: Promote the attributes analysts query frequently, and keep infrequent or volatile fields in the payload. That approach limits runtime cost while preserving fidelity for rarely used attributes. Teams should make the choice per engine and revisit it as query patterns, class coverage, and retention requirements change.
👉 Read our full editorial: OCSF lakehouse storage turns on engine choice, not schema width
