Iceberg stores sparse columns cheaply because absent values cost nothing in a file and schema changes are mostly metadata updates. ClickHouse pays a physical cost per column during inserts and merges, so very wide tables become expensive much faster. That difference is why the same OCSF schema can be fine in one engine and painful in another.
Why This Matters for Security Teams
OCSF adoption is not just a data modeling choice. It affects ingestion cost, schema evolution, incident triage speed, and how reliably security teams can retain fields across vendors and telemetry sources. Wide OCSF tables often look elegant in a design review, but the storage engine determines whether that elegance stays cheap or becomes operational debt. In Iceberg, sparse fields are mostly a metadata problem. In ClickHouse, wide physical tables can turn into insertion and merge overhead that compounds as pipelines grow.
This matters because security data rarely arrives in a neat, stable shape. New detections, enrichment fields, and vendor-specific attributes appear over time, and a schema that is easy to extend becomes much easier to govern. That is consistent with the control intent behind the NIST Cybersecurity Framework 2.0, where data integrity, governance, and operational resilience depend on choosing architectures that do not collapse under change. Practitioners often focus on query speed first and only later discover that schema width and sparsity have changed the cost model entirely. In practice, many security teams encounter this only after the first major telemetry expansion has already made storage and merge operations unstable.
How It Works in Practice
Iceberg and ClickHouse handle wide schemas differently because they are optimized for different storage behaviors. Iceberg separates table metadata from the underlying files, so adding columns is usually lightweight and missing values do not consume meaningful space in existing data files. That makes it well suited to OCSF-style telemetry where many fields are optional, vendor-specific, or only populated for certain event types.
ClickHouse is a columnar database, but it still pays a physical cost for each column during ingestion, compression, and merges. A very wide table increases write amplification, especially when many columns are mostly null or sparsely populated. That does not make ClickHouse a bad choice, but it means the data model should be shaped for the engine rather than copied blindly from a canonical schema.
- Use Iceberg when schema drift is expected and field-level sparsity is high.
- Use ClickHouse when the most important workloads are fast analytical queries over a smaller, better-curated subset of fields.
- Keep OCSF as a logical contract, not always as a literal physical table layout.
- Separate hot operational fields from rarely used enrichment fields when the engine cost rises.
For teams mapping this into a broader operating model, the key is to treat schema governance as a control plane concern, not just a database tuning exercise. The NIST Cybersecurity Framework 2.0 emphasizes governance and recovery as practical outcomes, which aligns with keeping telemetry adaptable as sources evolve. This guidance tends to break down when high-cardinality enrichment, frequent backfills, and heavy merge workloads all hit the same ClickHouse cluster because the ingestion path becomes the bottleneck before query design can compensate.
Common Variations and Edge Cases
Tighter schema discipline often improves query consistency, but it also increases modelling overhead, requiring organisations to balance operational simplicity against long-term flexibility. That tradeoff becomes visible when teams have to decide whether a field belongs in the canonical OCSF shape, a derived table, or an enrichment layer.
Best practice is evolving here. There is no universal standard for when a wide OCSF table should stay wide versus being normalized into domain-specific layers. In environments with stable, well-understood event types, ClickHouse can perform well if the table is curated and the number of actively populated columns stays moderate. In environments with frequent vendor onboarding, evolving detection content, or aggressive data retention requirements, Iceberg often fits better because metadata changes are cheaper than physical table rewrites.
The same logic applies when security teams want to preserve raw fidelity for investigations while still supporting analytics. A common pattern is to land raw or semi-structured OCSF data in Iceberg, then project a narrower analytical view for faster querying. That approach also reduces the risk of overfitting a warehouse design to one investigation workflow. For adjacent governance expectations around resilience and change control, the NIST Cybersecurity Framework 2.0 remains the clearest reference point for keeping the platform adaptable without losing control.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST CSF 2.0 provides the primary governance reference for this topic.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | GV.OC-01 | Schema choice affects data governance and operational resilience. |
Define telemetry storage decisions as governed architecture choices with clear ownership and review.
