Security teams should normalize, enrich, and transform telemetry before storage so the database receives consistent, queryable records. A vendor-neutral schema such as OCSF helps reduce schema sprawl, improves downstream queries, and makes security analytics easier to operationalize across logs from cloud, endpoint, proxy, and SaaS sources.
Normalize telemetry at the ingestion boundary, not inside ClickHouse
ClickHouse is best treated as the analytical store, not the place where messy source data gets reconciled. Security teams get better performance and more reliable searches when they standardize field names, timestamps, severity, actor identifiers, and event categories before loading data. That keeps the table design stable and prevents every dashboard or detection rule from inventing its own interpretation.
Normalization also makes it easier to preserve evidence quality. If one source emits telemetry pipeline structure with consistent redaction and enrichment, analysts can compare records across cloud, endpoint, proxy, and SaaS logs without repeatedly cleaning the same data at query time.
For high-volume security analytics, a sensible pattern is raw ingest, then parsing and enrichment into a curated schema, then materialized views or downstream tables for use cases such as detections, hunt queries, and reporting. That separation keeps the storage layer predictable while still allowing teams to evolve mappings as source formats change.
Use a vendor-neutral schema to control sprawl
A common schema is what stops telemetry from becoming a pile of one-off log formats. Vendor-neutral models such as OCSF help teams map heterogeneous sources onto a shared vocabulary, which reduces duplicate parsing logic and makes joins across product families more reliable. The value is not just cleanliness, it is operational consistency across sources that describe the same event in different ways.
This matters most when the same security question must be answered across many systems. A login failure, privilege change, or process execution event should land with the same core fields no matter whether it came from cloud control plane logs, an endpoint sensor, a firewall, or a SaaS audit trail. If the schema is stable, detections become easier to port and less brittle when a source changes its native format.
Schema discipline also helps with enrichment design. Decide which context belongs in the record itself, such as tenant, host, user, IP, or application, and which context should be derived later, such as asset criticality or identity group membership. That prevents overloading the ingestion path with enrichment that is better handled from reference data.
Design for queryability, retention, and downstream detection
Teams should structure records so the most common security questions are cheap to ask. That means choosing columns and types that support time slicing, filtering by source, grouping by actor or asset, and joining on stable identifiers. In practice, this is where good telemetry design improves both detection speed and analyst ergonomics.
It also means separating immutable evidence from derived analytics. Keep the original event content available where possible, but do not rely on raw blobs alone if you expect ClickHouse to power hunts, dashboards, and alert triage. A curated schema should carry the fields analysts use repeatedly, while preserving enough source detail to revisit the original event when needed.
For teams running large security datasets, open source SIEM and detection tools often expose the same design trade-off: the more consistent the upstream telemetry model, the less brittle the downstream detection logic becomes.
Risk and Threat Considerations
Raw logs that are loaded without normalization often create hidden security risk rather than analytical value. Inconsistent field mapping can conceal important activity, duplicated records can distort counts, and unredacted secrets or tokens can move from a source system into a long-lived analytics store where more people and more queries can see them.
Failure mechanism: Ingestion pipelines that skip schema normalization or redaction preserve source noise, but they also preserve source weaknesses, including malformed timestamps, inconsistent actor fields, and sensitive values that should have been removed before storage.
Impact: Detection logic becomes harder to trust, hunts become slower, and an analytics platform can itself become a repository of exposed credentials, personal data, or high-value operational context.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
CIS Controls v8, NIST CSF 2.0 and OWASP ASVS set the technical controls, while ISO/IEC 27001:2022 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| CIS Controls v8 | CIS-13 — Network Monitoring and Defense | Normalized telemetry supports central monitoring, correlation, and detection workflows. |
| Recommendation — Standardize telemetry fields so monitoring and detection can correlate events consistently. | ||
| NIST CSF 2.0 | PR.DS-01 — Data-at-rest is protected | Curated telemetry stores need controlled handling of sensitive log content before analytics. |
| DE.CM-01 — Networks and network services are monitored to find potential cybersecurity events | ClickHouse analytics depend on telemetry that is consistent enough for monitoring and detection. | |
| Recommendation — Apply data handling controls before loading logs into analytics storage. Normalize telemetry so monitoring detections can operate on comparable event records. | ||
| ISO/IEC 27001:2022 | A.8.15 — Logging | Telemetry structuring directly affects how logs are collected, prepared, and used for security monitoring. |
| Recommendation — Define log preparation rules that preserve useful evidence while standardizing records. | ||
| OWASP ASVS | V16 — Security Logging and Error Handling | The question concerns how security telemetry should be prepared for reliable analysis and investigation. |
| Recommendation — Structure events so security logging remains consistent, searchable, and investigation-ready. | ||
Practitioner Guidance
What to prioritise: Standardize the minimum security fields first, event time, source, actor, action, outcome, object, and environment, then add enrichment only where it improves correlation or triage. If a field is not stable across sources, do not make it mandatory in the first ingestion pass.
What to verify: Before trusting the pipeline, confirm that time zones, field names, severity mapping, and redaction rules produce the same meaning across all major source types. Also verify that query patterns against the curated tables are actually faster and simpler than querying raw events directly.
Practitioner takeaway: Treat ClickHouse as the place where normalized telemetry becomes usable intelligence, not the place where inconsistent source data gets cleaned up on demand.
Related resources from NHI Mgmt Group
- Should organisations require security telemetry before adopting SaaS tools?
- How should security teams structure crisis decision rights before an incident happens?
- How should startups structure security coverage before hiring a full team?
- Why do identity signals matter more than raw security telemetry?
Deepen Your Knowledge
Free weekly newsletter
Subscribe to the NHI & AI Identity Journal
The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.
Bonus 33% off our NHI Course when you subscribe.
Reviewed and updated by the NHIMG editorial team on October 30, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org