How should security teams structure raw telemetry before loading it into ClickHouse for analytics?

Why This Matters for Security Teams

Raw telemetry is rarely useful at scale if every source arrives with different field names, timestamps, severity labels, and entity identifiers. Before loading data into ClickHouse, security teams need a consistent structure that preserves investigative value while making search, aggregation, and correlation reliable. That matters because analytics quality depends less on storage speed than on whether the events can be joined across cloud, endpoint, identity, and SaaS activity without brittle parsing logic. The NIST Cybersecurity Framework 2.0 reinforces that observability and detection are not just tooling choices, but part of a broader risk management posture.

The common mistake is to ingest everything “as is” and defer normalization until a query or dashboard breaks. That approach usually creates duplicate pipelines, inconsistent enrichment, and expensive rework when analysts need to pivot quickly during an incident. A better pattern is to define a landing shape for raw events, then standardize the fields that matter most for detection, triage, and investigation. In practice, many security teams discover their telemetry design gaps only after an incident forces cross-source correlation that their schema cannot support.

How It Works in Practice

For ClickHouse, the goal is to make telemetry analytically efficient without destroying source fidelity. That usually means separating the pipeline into a raw capture layer, a normalized event layer, and, where useful, a detection-ready layer. Raw capture should retain the original payload or a minimally transformed copy for forensic traceability. The normalized layer should convert source-specific terms into a shared model, such as OCSF, so downstream queries can use stable fields instead of custom mappings for each product or service.

Key transformation steps usually include timestamp alignment, severity normalization, entity resolution, and enrichment with context such as asset inventory, user identity, cloud account, or threat intelligence. Security teams also need to decide which fields belong in structured columns and which remain in flexible payload fields. ClickHouse performs best when high-value dimensions are materialized cleanly, while volatile or sparse attributes stay in nested or semi-structured form.

• Standardize event time to a single timezone and precision before ingestion.
• Map source labels to shared categories for action, outcome, severity, and event class.
• Normalize host, user, IP, workload, and tenant identifiers for cross-source joins.
• Enrich only with context that improves search, detection, or response workflows.
• Preserve the original event or raw JSON for audit and reconstruction needs.

For operational guidance, the OWASP ecosystem is useful for thinking about data handling risks, while MITRE content helps teams model how telemetry supports adversary detection and investigation. Best practice is to treat schema design as a security control, not just an engineering preference. These controls tend to break down when high-cardinality cloud logs, rapidly changing SaaS fields, or untrusted parser output are forced into one rigid table without a clear normalization boundary.

Common Variations and Edge Cases

Tighter telemetry standardization often increases pipeline complexity and upfront engineering cost, so organisations must balance analytic consistency against speed of onboarding new sources. Current guidance suggests that the right balance depends on whether the data is needed for real-time detection, retrospective hunting, compliance reporting, or all three. There is no universal standard for every source type, especially where vendors emit nested objects, arrays, or highly dynamic fields that change without notice.

Edge cases show up when teams ingest proxy, DNS, EDR, SaaS, and cloud control-plane logs into the same analytics stack. Some sources are best normalized aggressively because their value comes from correlation, while others should be kept closer to raw form because investigators may need the exact original wording or sequence. This is also where identity context matters: if users, service accounts, and non-human identities are not resolved consistently, correlation in ClickHouse can miss privilege abuse, token misuse, or automation-driven activity. A practical rule is to normalize what supports detection logic and preserve what supports evidentiary integrity.

For teams operating under broader resilience expectations, alignment with identity, logging, and control objectives in NIST CSF or related governance programmes is more useful than chasing a one-size-fits-all schema. Where the environment includes agentic workflows or automated responders, telemetry should also capture the acting identity, the tool invoked, and the triggering context so analysis can distinguish human action from machine-executed activity.

Related resources from NHI Mgmt Group

• How should security teams structure crisis decision rights before an incident happens?
• How should security teams implement telemetry pipelines before the SIEM?
• How should security teams structure an IAM programme before selecting technology?
• How should security teams move high-volume telemetry into a data warehouse without losing structure?