OCSF lakehouse storage turns on engine choice, not schema width

At a glance

What this is: This analysis shows that OCSF storage strategy should be chosen by engine behavior, with Iceberg and ClickHouse favouring different table layouts.

Why it matters: It matters to security data engineers and SOC teams because storage choices affect query performance, schema evolution, and the reliability of downstream detections built on OCSF data.

By the numbers:

• Altinity measured 64 GB of peak RAM and 167 seconds to insert one million rows into a 10,000-column ClickHouse table.
• 6.5 GB of peak RAM and 12 seconds
• Iceberg only infers column metrics for the first 100 columns by default, so manifest breadth can become the real cost of wide schemas.

👉 Read TENZIR's analysis of OCSF storage strategies in Iceberg and ClickHouse

Context

OCSF solves event normalisation, but it does not solve storage design. Once normalized events arrive, teams still have to decide whether to build one wide table, split by class or source, or keep a promoted-column model with semi-structured spillover. That decision becomes especially important in OCSF lakehouse architectures, where schema shape and engine behavior jointly determine whether the design stays manageable.

The core identity and security governance lesson is that ingestion architecture is now part of operational control. A storage layout that is harmless in one engine can create metadata pressure, merge overhead, or query-planning drag in another, which is why security data pipelines need explicit design choices rather than inherited warehouse habits. For teams handling security telemetry at scale, this is a data engineering problem with downstream implications for detection and auditability.

Key questions

Q: How should security teams choose an OCSF storage layout for different engines?

A: Start with the engine, not the schema. Iceberg can handle wide, sparse OCSF tables efficiently because the main cost is metadata, while ClickHouse usually performs better when hot fields are promoted and the long tail stays in semi-structured storage. The right choice depends on query patterns, file mechanics, and schema evolution behavior.

Q: Why do wide OCSF tables work better in Iceberg than in ClickHouse?

A: Iceberg stores sparse columns cheaply because absent values cost nothing in a file and schema changes are mostly metadata updates. ClickHouse pays a physical cost per column during inserts and merges, so very wide tables become expensive much faster. That difference is why the same OCSF schema can be fine in one engine and painful in another.

Q: What breaks when OCSF is modelled as one giant table in the wrong engine?

A: The main failure mode is not schema correctness but operational cost. In ClickHouse, insert memory, merge time, and file counts rise with each additional column, while in Iceberg the risk shifts toward metadata bloat and slower planning. If teams ignore that distinction, the data layer becomes harder to query and harder to maintain.

Q: How can teams decide whether to use promoted columns or semi-structured payloads?

A: Promote the attributes analysts query frequently, and keep infrequent or volatile fields in the payload. That approach limits runtime cost while preserving fidelity for rarely used attributes. Teams should make the choice per engine and revisit it as query patterns, class coverage, and retention requirements change.

Technical breakdown

Why OCSF maps cleanly to JSON but not to one storage model

OCSF defines a normalized event structure, but storage engines interpret that structure very differently. In Iceberg, the file format and metadata model make wide sparse schemas relatively cheap because absent columns cost nothing in a file and schema evolution is metadata-only. In ClickHouse, every physical column has real storage, insertion, and merge costs, so the same schema becomes expensive as column counts rise. The practical result is that OCSF is a contract for event shape, not a prescription for table topology.

Practical implication: choose table topology after you choose the engine, not before.

Iceberg wide tables versus metadata hygiene

Iceberg with Parquet changes the economics of width. Null-heavy columns compress well, hidden partitioning avoids helper columns, and adding fields does not require rewriting historical files. The constraint is metadata, not raw storage. Per-file statistics and manifest breadth can slow planning, especially when column counts grow and metrics coverage expands. That makes schema width a governance issue about metadata hygiene and file pruning efficiency rather than a basic storage penalty.

Practical implication: keep an eye on manifest growth and stats coverage, not just file size.

Why ClickHouse pushes teams toward promoted columns and payload spillover

ClickHouse behaves like a shared-nothing OLAP engine, so wide sparse tables are costly at insert time and during merges. The platform creates a physical cost for each column, and Nullable fields add extra mask overhead. That is why production SIEM-style deployments often promote the hottest fields into typed columns while retaining the long tail in JSON. The design keeps query speed acceptable without forcing every rare attribute into the same expensive column footprint.

Practical implication: promote only the fields you query often and leave the rest in payload storage.

NHI Mgmt Group analysis

Engine-aware OCSF design is now a governance choice, not a storage detail. OCSF gives teams a shared event vocabulary, but it does not tell them how to organise data for analysis. That omission matters because the wrong table strategy can create metadata drag, query latency, or operational complexity that undermines telemetry value. Practitioners should treat storage layout as part of security data governance, not a downstream implementation detail.

Iceberg makes wide schemas viable, but only if metadata remains disciplined. The article correctly separates file cost from manifest cost, which is where many warehouse habits fail when moved into lakehouse designs. That means the real control variable is not how many columns exist in theory, but how carefully teams manage statistics, partitioning, and schema evolution. Practitioners should optimise for Iceberg metadata health, not just data-file efficiency.

ClickHouse forces a different truth: sparse width is a runtime tax. The performance penalties described here show why promoted columns plus JSON spillover have become a common operational pattern. This is a useful reminder that a storage model can be acceptable in one analytics engine and untenable in another, even for the same OCSF data. Practitioners should align schema strategy to the execution engine they actually run.

OCSF lakehouse architecture introduces a new concept: schema portability debt. Teams that assume a table layout can move unchanged between warehouse and lakehouse engines accumulate hidden cost when sparsity, statistics, and merge behavior differ. The article shows why the same logical schema can create different operational outcomes. Practitioners should design for engine-specific portability constraints instead of assuming one universal OCSF layout.

This is adjacent to identity governance because telemetry quality shapes access accountability. When security events are stored poorly, investigations into privileged access, service accounts, and administrative actions become slower and less reliable. That does not make OCSF an IAM article, but it does mean identity evidence depends on how well the data layer preserves fidelity. Practitioners should treat security telemetry design as part of audit readiness.

What this signals

OCSF storage design will increasingly be evaluated as part of the security data pipeline, not as a neutral engineering choice. For teams managing privileged activity, service-account telemetry, or authentication evidence, the difference between a fast query path and a sluggish one can affect investigation quality and audit confidence.

Schema portability debt: teams that assume OCSF layouts can move unchanged between engines will keep rediscovering hidden cost in metadata, merges, and planning. The better pattern is to define storage rules per execution engine and make them explicit in the pipeline specification.

For identity-heavy telemetry, the practical question is whether your data model preserves enough fidelity to support access reviews, incident triage, and forensic reconstruction. That makes security data engineering part of identity governance in practice, even when the topic looks like pure infrastructure architecture.

For practitioners

• Match table strategy to the storage engine Use a wide nested table in Iceberg when schema evolution and sparse columns are the norm, but prefer promoted columns with payload spillover in ClickHouse-style engines.
• Cap metadata growth before it affects planning Track manifest size, column statistics coverage, and partition-spec changes so width does not degrade Iceberg query planning.
• Promote only hot fields in OLAP pipelines Keep high-frequency attributes as typed columns and leave rare attributes in a semi-structured payload so insert and merge costs stay controlled.
• Treat OCSF routing as an ETL design decision Document the routing key, table topology, and schema-evolution rules in the pipeline itself so data shape changes remain intentional and auditable.

Key takeaways

• OCSF normalises event structure, but the storage engine still determines whether width is cheap or costly.
• Iceberg and ClickHouse reward different table topologies, so schema design must be engine-specific.
• Security telemetry pipelines should treat metadata hygiene, promotion rules, and query patterns as governance controls.

Key terms

• Open Cybersecurity Schema Framework: A standard vocabulary for representing security events across tools and platforms. OCSF helps teams exchange telemetry in a consistent way, but it does not replace the need for enrichment, correlation, or behavioural analysis inside the security stack.
• Iceberg metadata planning: The part of Iceberg that decides how files, partitions, and column statistics are tracked for efficient queries. It matters because wide schemas are usually limited by manifests and statistics, not by raw file storage, so metadata discipline becomes a performance control.
• Semi-structured payload strategy: A storage pattern that promotes frequently queried fields to typed columns while preserving the rest of the record in JSON or another flexible payload. It balances query speed with schema flexibility, especially in engines where every extra column has a measurable runtime cost.
• Schema portability debt: The hidden operational cost that appears when a table design built for one analytics engine is reused in another with different storage economics. It shows up as unexpected performance loss, maintenance complexity, or metadata growth when the same logical schema meets different execution behavior.

What's in the full article

TENZIR's full article covers the operational detail this post intentionally leaves for the source:

• The exact OCSF table strategies compared across Iceberg and ClickHouse implementations
• The to_iceberg operator behavior for table creation, partitioning, and schema evolution
• The practical mechanics of Parquet file sizing, stats collection, and hidden partitioning
• The ClickHouse promotion pattern used by production SIEM-style deployments

👉 The full TENZIR article covers the engine-specific trade-offs, OCSF table layouts, and ingestion blueprint in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and lifecycle controls. It helps practitioners connect identity governance to the broader security architecture decisions that shape real operational risk.